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Preface 


The goal of this textbook is to support the teaching of digital and statistical signal 
processing in higher education. Particular attention is paid to the presentation of the fun- 
damental theory; key topics are outlined in a comprehensible way, and all areas of the 
subject are discussed in a fashion that aims at simplification without sacrificing accuracy. 

The book is divided into two sections. 

In the first section, we aim at a deep understanding of the subject of Digital Signal 
Processing and provide numerous examples and solved problems often dealt with the use 
of MATLAB®. 

The second section covers Statistical Signal Processing. The basic principles of statistical 
inference are discussed and their implementation in practical signal and system condi- 
tions are analyzed. The discussion is strongly supported by examples and solved prob- 
lems in this section, as well. 

The content of the book is developed in ten chapters and two appendices as follows: 

Chapter 1 contains introductory concepts for a basic understanding on the field of 
Digital Signal Processing, with particular emphasis on Digitization and Reconstruction of 
continuous-time signals. 

Chapter 2 refers to the characteristics and properties of discrete-time signals and 
systems. 

Chapter 3 refers to z-Iransform, which is a basic mathematical tool for studying 
discrete-time systems. 

Chapter 4 analyzes the implementation of the filters FIR and IIR in various forms (Direct 
Form I and II, Cascade form, Parallel form). 

Chapter 5 describes the Analysis of Discrete Systems in the Frequency domain. We 
analyze the Discrete Time Fourier Transform, Discrete Fourier Series, Discrete Fourier 
Transform, Fast Fourier Transform and Discrete Wavelet Transform. 

Chapter 6 deals with the design of the IIR and FIR digital filters. Indirect design meth- 
ods (Invariant Impulse Response, Invariant Step Response, Differential Imaging, Bilinear 
Transform, Pole and Zero Position Matching) and direct methods of designing IIR filters are 
analyzed. The design of FIR filters through sampling in the frequency domain and with the use 
of Windows (Rectangular, Bartlett, Hanning, Hamming, Blackman, Kaiser) is also described. 

Chapter 7 describes comprehensively and develops the most important statistical mod- 
els used in stochastic signal processing. 

Chapter 8 deals with the concept of parametric estimation of stochastic signals when 
processed for both one- and multi-dimensional cases. 

Chapter 9 analyzes the particular problem of linear estimation. This problem is wide- 
spread in many practical applications due to its attractively low computational complexity 
in processing stochastic signals. 

Chapter 10 hosts the fundamental principles of stochastic signal detection. The most well- 
known theoretical approaches and practical techniques for detecting signals are analyzed. 

Appendix I summarizes the basic principles in the Algebra of Vectors and Matrices, 
with a particular emphasis on their application to stochastic signals and systems. 

Finally, Appendix II presents numerous solved problems in Statistical Signal Processing, 
aiming at further understanding and consolidation of the theoretical concepts of the sec- 
ond section of the book. 
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Introduction 


1.1 Introduction 


A signal can be defined as a function that conveys information about properties, such as 
the state and behavior of a physical system. The signal contains this information in a form 
of a certain type of variation, independent of signal representation. For example, a signal 
can develop temporal or spatial variations. Signals can be represented as mathematical 
functions of one or more independent variables. Signals in which the variables of time 
and amplitude take discrete values are defined as digital signals. These signals can be 
described by a sequence of numbers. 

The field of digital signal processing is an essential part of applied science and engineering 
since much of the information of our world is, today, being recorded in signals, and signals 
must be processed to facilitate the extraction of information. Consequently, the develop- 
ment of processing techniques is of great importance. These techniques are usually based 
on the application of transforms that convert the original signal to other forms, which are 
more appropriate for processing and analyzing. 

The ability to proceed to digital implementation of modern signal processing systems 
made it possible for the field to follow the general trend of cost and size reduction and to 
increase the reliability of all digital systems. Digital signal processing has a vast number of 
applications in many different fields of modern science, such as medicine, telecommunica- 
tions, geophysics, image processing, automatic control systems, etc. 

In general, digital signal processing has the following objectives: 


e Improvement of image quality 
¢ Separation of undesired component in the signal 
¢ Signal compression 


Search for periodicity 


Deconvolution of input system 
e Recognition of the status of the system 


Since the system conveys information about a physical quantity, then, from a mathemati- 
cal point of view, it can be considered as a function. We will assume that our signal is a 
sequence of numbers in the memory unit of a computer of general purpose or of a special- 
ized digital device. 

Most of the signals that we encounter in theory are in analog form. This is because sig- 
nals are functions of continuous variables (such as time), and their values are usually ona 
continuous scale. These signals can be processed by suitable analog systems such as filters, 
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Digital signal processing. 


frequency analyzers or frequency multipliers, in order, for example, to have their charac- 
teristics changed. In such cases, we say that the signals have been processed, as shown in 
Figure 1.1, where the input and the output signal are in analog form. 

Digital signal processing provides an alternative method of processing an analog signal, 
as shown in Figure 1.2. 

In order to implement digital processing, there is a need for an interface between the 
analog signal and the digital processor. This interface is called an analog to digital converter 
(ADC). The output of the converter is a digital signal, which, in turn, is the input to the 
digital signal processor. 

The digital signal processor may be a computer or a microprocessor, which is pro- 
grammed to perform the desired operations on the input signal. In applications where the 
digital output from the digital signal processor is to be given to the user in analog form, 
as, for example, in speech communication, another circuit has to be mediated between the 
digital and the analog region. A circuit that receives a digital input and converts it into 
analog is a digital to analog converter (DAC). A key feature of DACs is the accuracy in which 
the converter can change the output. This accuracy is a function of the number of the 1 bits 
(2" levels of quantization) that the digital circuit of the converter can operate at. 


1.2 Advantages of Digital Signal Processing 


There are many reasons why digital processing of an analog signal is common practice. 


e First of all, a programmable digital system allows the user to easily change the 
parameters of digital signal processing simply by modifying the program. 
Changing functions in an analog system usually requires redesigning the hard- 
ware and testing its proper operation. 
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e The accuracy of the desired results plays an important role in determining the 
form of the signal processor. Digital signal processing provides better control of 
the accuracy required, whereas the tolerance of an analog signal processing sys- 
tem makes it very difficult for the system designer to control its accuracy. 


e Digital signals are easy to store on magnetic media without wear or loss of signal fidel- 
ity, so the signal can easily be transferred and processed in another laboratory space. 


e The digital signal processing method allows the use of more sophisticated signal 
processing algorithms. It is usually very difficult to make accurate mathematical 
operations in an analog form. However, all these operations can be done easily by 
a computer. 


e In some applications, digital signal processing costs less than analog. The lower 
cost may be due to the fact that digital equipment is cheaper, or is the result of the 
simplicity of adjustment of the digital application. 


As a result of the aforementioned advantages, digital signal processing has been applied 
to a wide range of disciplines. For example, applications of the digital signal processing 
techniques include speech processing, signal transmission to telephone channels, image 
processing, seismology and geophysics, oil exploration, nuclear explosions detection, pro- 
cessing of space signals, etc. 

However, digital signal processing has limits, too. A practical limitation is the speed 
of ADCs and digital signal processors. Moreover, there are signals with a wide range of 
frequencies, which either require very fast ADCs and digital processors or whose digital 
processing is beyond the capacity of any digital equipment. 


1.3 Digitization Steps of Analog Signals 


In digitization, an analog signal is converted to digital (usually binary), i.e, from a continuous 
function of time, the analogue signal is converted to a function of a series of discrete numeri- 
cal values. The process of digitization of continuous time signals involves the following steps: 


¢ Sampling 
¢ Quantization 
¢ Coding 


1.3.1 Sampling 


Sampling is the process in which a physical quantity that varies in time or space is depicted 
as a set of values at discrete locations (samples) in the domain of the physical quantity. The 
grid of sampling points is usually selected uniformly (uniform sampling). The set of samples 
is the discrete-time (or space) signal. In Figure 1.3, an analog signal x(f) (1.3(a)) and the 
corresponding discrete time signal x,(f) resulting from the sampling of the analog signal 
(1.3(b)) with a sampling frequency, f, are illustrated. 

We define sampling period, T,, as the distance (in sec) between two successive samples 
and as sampling frequency, f, = 1/T,, the number of samples per second. The signal after 
sampling is not the same with the original since we only maintain the x(kT7s) values and 
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FIGURE 1.3 
(a) Analog signal, (b) the signal after uniform sampling. 


reject all other values. The fidelity of the discrete time signal depends on the sampling 
frequency, f,. In the case of low sampling frequency, rapid changes in the original signal 
are not recorded, and the information contained in it is lost, resulting in degradation of the 
original signal. The problem is corrected by increasing the sampling frequency. Therefore, 
the value of the sampling frequency must meet two conflicting requirements: 


e High digitization quality (high sampling frequency required) 
e Digital files of a small size (low sampling frequency required) 


In many practical applications of digital signal processing, the usual problem of the 
varying sampling rate of the signal often appears. One way to implement varying sam- 
pling rate is to convert the sampled signal to its original analog form and then sample 
again. Another way is to make the signal undergo new sampling in its digital form, with 
the advantage that no additional distortion is introduced by the successive processing of 
the signal by additional DACs and ADCs. 

The conversion of sampling rate digitally is being implemented as follows: 


1. By reducing the sampling rate by an integer factor, e.g., M. Therefore, the new sampling 
period is T. = MT, and the re-sampled signal is x,[”] = x,[nT. ] = x,[nMT,] = x[nM]. 
The procedure of reducing the sampling rate by an integer factor, M, is done by 
picking out each M-th sample of the sequence x[n]. The system that performs this 
procedure is called downsampling. Downsampling generally results in overlap- 
ping, which can be avoided if the signal, x[n], is filtered prior to the process by 
means of a low-pass filter with a cut-off frequency, w, = 2/M. 


2. By increasing the sampling rate by an integer factor, e.g., L. In this case, it is necessary 
to extract the samples x,[n] = x,[nT/L] from x[n]. The samples x,[n] for the values of 
n, which are integral multiples of L, are derived from x[n] as follows: x,[nL] = x[n] 
The sequence, x[n]. The upsampler extends the time-scale by a factor of L by adding 
L-1 zeros at the intervals between the samples, x[n]. 


3. By converting the sampling rate by a factor that is a rational number. A link-in-series 
decimator, which reduces the sampling rate by a factor, M, with an interleaver, 
which increases the sampling rate by a critical factor, L, leads to a system that 
changes the sampling rate by an rational coefficient, L/M. 
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Example: 


Suppose that an analog signal x,(f) is sampled at a frequency of 8 KHz. The purpose 
is to obtain the discrete-time signal that would have been generated if x,(f) had been 
sampled at a frequency of 10 KHz. What is asked is to change the sampling rate by a 
factor, ue aa This is done by multiplying its sampling frequency by a factor of 
5, filtering the frequency-amplified signal through a low-pass filter with a cut-off fre- 
quency, w, = 2/5, of gain 5, and, finally, dividing the frequency of the filtered signal by 
a factor of 4. 


1.3.2 Quantization 


Quantization is the conversion of continuous amplitude values into discrete amplitude val- 
ues (levels). A continuous signal has a continuous amplitude domain and therefore its 
samples have a continuous amplitude domain, too. That is, within the finite signal domain 
we find an infinite number of amplitude levels. Sampling makes the analog signal discrete 
but if the values of the samples are to be represented in a computer, the amplitude values 
must also be discrete, i.e., they must be approximated by other predetermined values. The 
difference between the actual value of the sample and the value finally encoded is called 
a quantization error (Figure 1.4). 

This difference introduces a distortion to the quantized signal which can be dimin- 
ished if more quantization levels are used. This, however, would require the use of 
more bits in order to encode the sample. During the quantization process, the range 
of amplitude values of the continuous signal is subdivided into specific levels, and a 
digital code is assigned to each level. Each sample gets the digital value closest to its 
original value. The number of available levels depends on the number of digits of the 
code, n (sampling size), used to represent these values. The number of encoding levels 
is equal to 2". 

Figure 1.5 illustrates the relation between the analog input and digital output for a three- 
input ADC or quantizer. The horizontal axis represents the analog input of a range from OV 
to 10V. The vertical axis represents the encoded binary output, and, in the case of a 3-digit 
code, it increases from 000 to 111. In this case we have 2"-1 analog points. In the case of a 
3-digit quantizer, these analog points are 0.625, 1.875, 3.125, 4.375, 5.625, 6.875 and 8.125V. 
The central points of the encoded output words are 1.25, 2.50, 3.75, 5.00, 6.25, 7.50 and 8.75V. 


Quantization error 


-3ts  -2ts  -1ts ts 2ts 3ts 4ts 5ts 6ts 7ts — 8ts n 


FIGURE 1.4 
Quantization error. 
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FIGURE 1.5 
The input-output relation of one ADC 3 digits. 


The quantization level is the step between two successive reference points and is given 
by the formula Q = FSR/2" where the FSR is the Full Scale Range. If the FSR is 10V, and 
n = 3, then Q equals 1.25V. 

Graphically, the quantization process means that the straight line representing the rela- 
tion between the input and output of a linear continuous system is replaced by a stair-step 
graph. The difference between the two successive discrete values is called step-size. The 
signals applied to a quantizer are graded at amplitude levels (steps), and all the input sig- 
nals that are at the plus or minus half of a level are replaced at the output by the mid-value 
of this level. 

Since the number of digits of a digital word is finite, the result of A/D conversion is finite. 
This is because the digital output must have only a finite number of levels, so an analog 
number should be rounded to the nearest digital level. Consequently, any A/D conversion 
contains a quantization error. Such errors range between 0 and +%Q. Quantization error 
depends on the accuracy of the quantization level and can be as small as it is desired, by 
increasing the number of digits, 1. In practice, there is a limit to the number of the digits, 
n, used, and that is why there is always an error due to the quantization. The ambiguity 
that occurs during quantization process is called quantization noise. The quantization noise 
range is a significant measure for the quantization error because it is proportional to the 
average power associated with noise. The quantization error is usually described statisti- 
cally, and the quantization noise is a time-invariant stochastic process, i.e., its statistical 
description does not change over time, and the noise probability density function is uni- 
formly distributed over the quantization error range. 

The quality of the quantized signal is measured by the signal-to-noise ratio (SNR = 6N + 
1.76 (dB) - where N is the number of bits). Each bit that is added to the signal analysis 
increases the SNR to 6dB. The resolution, most commonly used for music and speech pro- 
cessing, is 16 bit and gives a SNR of 96 bit. The numerical values that will be given by the 
signal quantization process during digitization depend on the electrical characteristics of 
the converter as well as the analog signal. For optimal digital conversion, the voltage of 
the analog signal should be within the limits of the input voltage of the converter. This is 
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usually accomplished by the proper adjustment of the input signal voltage through pre- 
amplification. Ideally, we use the entire dynamic range of the converter. If the input signal 
is weak due to fixed SNR, potential amplification in the digital system will also boost 
noise. If the signal is too strong and its electrical characteristics exceed the endurance of 
the converter, its extreme values will be quantized to the maximum (and minimum) value 
given by the specific resolution. In this way, inevitable loss of information of the original 
signal is introduced, which is called clipping distortion. 

The quantization process, as described above, uses a uniform distance between the 
quantization levels. In some applications, it is preferable to use a varying distance between 
the quantization levels. For example, the range of voltages covered by voice signals, from 
the maxima of loud voice to the minima of low voice, is in the order of 1000 to 1. Using 
a non-uniform quantizer (with the characteristic that the step size increases as the dis- 
tance from the beginning of the input-output amplitude axes increases), its last big step 
can include all possible fluctuations of the voice signal at high amplitude levels, which 
occur relatively rarely. In other words, ‘weaker’ steps, which require more protection, are 
preferred at the expense of ‘stronger’ ones. In this way, a percentage of uniform accuracy 
is achieved in most of the domain of the input signal, requiring fewer steps than those 
needed in the case of a uniform quantizer. 


1.3.3 Coding 


Coding is the representation of the quantized amplitude values of the signal samples in a 
bit-sequence, whereby the digital signal is generated as a series of bits. The code of each 
level is called a codeword, and the set of different codewords used for the encoding proce- 
dure is called code. A code is of fixed length when the set of codewords has the same size. 

The main disadvantage of the digital representation of continuous media is the distor- 
tion introduced by the sampling and quantization process. On the one hand, by ignoring 
some analog signal values, information is lost, and on the other hand, the approximation 
of the actual value of the signal with one of the available levels always contains a certain 
amount of error. This distortion decreases as the sampling frequency and the length of 
the word increases. However, in this case, the volume occupied by the information also 
increases, and consequently larger and faster storage media, as well as faster processing 
units, are required. 

Figure 1.6 illustrates the process of analog-to-digital signal conversion. 


Quantized signal 


A 
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Digital data 
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FIGURE 1.6 
Analog-to-digital signal conversion. 
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FIGURE 1.7 
Digital voice coding system. 


An example of the aforementioned process is voice encoding. This is a very useful pro- 
cess if we consider that a great amount of information is transmitted through voice and 
that, by compressing it, a significant reduction of the bandwidth required can be achieved 
(up to 4-5 times without noticeable quality degradation). In mobile telecommunication 
systems, bandwidth availability is an important resource as great effort is being made to 
serve more and more users within it. For this reason, complex voice coding algorithms 
are used to reduce the bandwidth of a signal without compromising quality. Most of the 
well-known voice coding algorithms are explicitly based on a voice production model. It 
is necessary to construct voice encoders that compress voice in limited bandwidth without 
sacrificing quality. 

All the original parts of a digital voice coding system are shown in Figure 1.7. The input 
of the system is a continuous sound waveform, s(t). This signal is filtered at low frequen- 
cies using an anti-aliasing filter and is sampled by an ADC to give the digital voice signal 
s(n). This is the input of the encoder. Voice encoding generally consists of three parts: voice 
analysis, parameter quantization and parameter encoding. The input, at the stage of analy- 
sis, is the digital voice signal, while the output is the new representation of the voice signal, 
which will be quantized and encoded. The output at this stage may vary greatly depend- 
ing on the way the voice signal is modeled. Quantization is used to reduce the information 
flow of the voice signal. The output of the quantizer is led to the encoder, which matches 
a binary code to each possible quantized representation. These binary codes are placed 
together for more efficient transmission or storage. 


1.4 Sampling and Reconstruction of Sinusoidal Signals 


Sampling is the procedure of obtaining signal values at selected instances. In the case of 
uniform sampling, these points are equidistant, and the distance between them is called 
the sampling period, T,. At the sampling process, there is a demand of minimum amount 
of information loss of the continuous time signal. 
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Let us consider that, at the input of an ADC, there is a sinusoidal signal of the form 
x(t) = Acos(wt + ). The signal at its output will be of the form x[n] = A cos(wnT, + y) =A 
cos(Qn + g). Consequently Q = wT, or F = f -T, = ca 

This relation correlates the relative frequency, which expresses the regression rate of the 
signal x[n], with the natural frequency of x(t) through the sampling frequency, f, = 1/T,. 

The sampling frequency f, must be chosen to satisfy the following relation: 


feiss ees (1.1) 


What Equation 1.1 describes is that when a sinusoidal signal is sampled, it must have a 
frequency less than half of the sampling frequency. Namely, the presence of at least two 
samples per cycle is necessary in order to avoid losing the information of the frequency 
value of the original signal. 

A continuous-time signal, x(t), with a maximum frequency, f, 


maxr CAN be successfully 
reconstructed from its samples, provided that: 


ina = & (1.2 (a)) 


or f, 22 fimax (1.2 (b)) 


Equations 1.2 (a) and 1.2 (b) are the mathematical expressions of the sampling theorem 
(Shannon, 1948). Frequency f,/2 is called cut-off frequency or Nyquist frequency. If we choose 
to work with sampling frequencies f,, that do not satisfy Shannon’s criterion, then the origi- 
nal spectrum will not be completely recovered, and effects, such as aliasing or folding, will 
corrupt it significantly. Examples, proof and a detailed discussion of the sampling theorem 
are presented next. 


Example 1: 


If we digitize a signal of a 30 KHz frequency using a sampling frequency f, = 50KHz, 
then we will get the folding effect, and the output signal will have a f = 50-30 = 20 KHz 
frequency. A sampling frequency greater than the Nyquist frequency is a waste of stor- 
age space since it creates additional samples without them being necessary for the suc- 
cessful reconstruction of the signal. 


Example 2: 


In case we want to transmit a speech signal, the frequency range 0-4000 Hz is suffi- 
cient. Therefore, we use a low-pass filter that allows the frequencies 0-4000 Hz to pass 
while cutting the remaining frequencies. The maximum frequency of the signal will be 
finax = 4000Hz. According to the sampling theorem, we will gain all the information con- 
tained in the spectrum of a conversation session if we transmit at least 8000 samples/sec 
of this spectrum. Therefore, the sampling frequency should be 8000Hz. 
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1.4.1 Proof of the Sampling Theorem and a Detailed Discussion 


Consider the model of ideal sampling with the use of impulses, as shown in Figure 1.8, 
where the signal x(t) is to be sampled and s(t) is the sampling impulse train of infinite 
duration. 


The signal x(t) is band limited in [-B,B], so the range of the spectrum is 2B. The signal 


s(t) that has the following spectrum S(f)=F {s(t)} =F » d(t - mt, is periodic with a 
period T,, and in order to compute its Fourier transform, we have to evaluate the coeffi- 
cients, C,,, of its Fourier series, namely 


s(t)= 1 att nt.) = YC, with a, = 0, = = 


n=—eo m=—co 


and 
Ty 
i 1 f 1 
Ca aa | stem d= | a(em at = — 

T. T. T. 

s (T,) s ee Ss 
2 
Thus, 
1 
C,=—VmeZ 
and 
: jm@ot 1 . jm@ot 
s(t) aa C,,€ = tT e 
m=—09 W n=-00 
Xs(t) 
(A is Xs(t)=x(t)s(t) 
x(t 
© | 
[tr 
> stt) Ts Ts 2Ts t 
Ix(thla | 
> 
-B k—>| B t | <>| 
28 7 
FIGURE 1.8 


Ideal sampling using impulses. 


Introduction 13 


The Fourier transform of s(t) would be: 


sin=Fao=F|t Sele LD Flot) 


m=—e0 ° m=—00 


= EY KF-m f=2¥5{ f-m 7] 


m=—oo fom fe m=—eo 


The result is illustrated in Figure 1.9. 

For the signal x,(f), resulting after the sampling process, it is true that x,(f) = x(£)s(t), so it 
can be assumed that it is derived from the multiplication of the original signal, x(t), and 
the signal s(t) (Sampling function), which controls the state of the switch, as illustrated in 
Figure 1.10. 

The spectrum of x,(¢) can be estimated as follows: 


X.(f) =F {x,()} = F {x(t)-s(t)} = F{x(p)} * F{s(t)} = 


xinesin=x(pe| > a[s-n2 = 


8 m=—co 


x= FY X( fmt | 


m=—99 


The spectrum |X,(f)| of x,(é is illustrated in Figure 1.11. 
As shown in Figure 1.11, it is obvious that, in order to avoid overlap between the suc- 


cessive copies of the original spectrum, X(f), around the center frequencies, mf, = Ma 
the minimum distance between the central frequencies (i.e., f) must be at least equal to the 
spectral range 2B. That is, we must apply the rule f, > 2B. In this case, the spectrum of the 
original signal is contained completely in the spectrum of the sampled signal and can be 
easily isolated using a low-pass filter without any distortion. 


-1/Ts 1/Ts 2/Ts t 


FIGURE 1.9 
Signals s(t) and S(f). 
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S(t) 


ELECTRONIC SWITCH 


X(t) Xs(t) 


FIGURE 1.10 
Sampling system. 


+ [Xs(f)| 


| X(f-fs) | IX(f)| | X(f-fs) | 


FIGURE 1.11 
Spectrum IX, (f) of x,(2). 


Under the circumstances that the above condition is not satisfied, there will be an over- 
lap between successive copies of the spectrum, X(f). The effect of overlap has the following 
result: the original spectrum, X(f), cannot be completely recovered from the spectrum, 
X,(f), and consequently, the original signal, x(é), from the signal, x,(f). In this case, the spec- 
tra of the various components at frequencies 0, +f,, +2f, ..... overlap, resulting in the distor- 
tion of the spectrum of the original signal. So, after the spectrum of the original signal is 
truncated by a filter, frequencies from the neighboring spectral component are still present 
within its range, too, resulting in the aforementioned spectrum distortion. In Figure 1.12 
we notice that, due to the choice of sampling frequency, f, < 2B, the upper spectral region 
of X(f) is distorted. 

The inverse process of sampling, which aims to the recovery of the original signal, is 
called reconstruction of the signal. From the spectra of the original and the sampled sig- 
nal, it becomes clear that the first can only be completely recovered when all frequencies, 
which are greater than f,,,, in the sampled signal, are cut off. 

Generally, the truncation of the spectrum X(f), and consequently of the original signal, 
x(f), is implemented by using a low-pass filter around the central spectrum, which cuts 
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FIGURE 1.12 
Spectrum X,(f) with overlaps. 


off the ‘remaining’ spectra and preserves only the central part, with a cut-off frequency 
f= bs > B (1.3), as illustrated in Figures 1.13 (a), (b) and (c). 


Thus, in order for the original signal to be recovered, it is necessary that there is no over- 
lap between the copies of the original signal spectrum in the spectrum of the sample. This 
only happens when the criterion f, > 2f,,,, of the sampling theorem is satisfied. 

The ideal low-pass filter, with a cutoff frequency f, = B, has a frequency response and an 
impulse response given by the Equations 1.4 and 1.5, respectively: 


t,, |fs# 

H(f)= ; (1.4) 
0, Ifl> 

h(t) =F {H(f)} = sin C +) (1.5) 


Consequently, the output of the low-pass filter is determined by using the convolution 
integral as follows: 


x(t) = x, (t) *h(t) = Y: xn )6(t= nT) vsine{ 27] 


n=—co 8 


x(0) 


co 


x(t) = >» x(nT.) sinc ze = WT.) (1.6) 


n=—0o 
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Xs(t) LPF. A 


FIGURE 1.13 
(a) Spectrum, (b) low-pass filter, (c) impulse response of the ideal low-pass filter. 
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In the case that we choose T, = a = gh we have: 


f, 2B 
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x(t) = y, o( 2 Jsine| ae 1 - n)| , £,=2B (1.7) 


n=—co 


Equations 1.6 and 1.7 are the interpolation relations through which we can achieve the 
recovery of the original continuous signal, x(), from its samples, x(nTs) Equation 1.7 states 
that ideally an analog signal (at infinite points in time) can be reconstructed when its 
samples x(nT,) are linearly combined with the function sinc. 


1.5 Physical Sampling 


The discussion in previous paragraph is only theoretical, since in practice there are several 
problems, namely: 


I. 


N 


There is no such signal (containing low harmonics) for which X(f) = 0, when|f|> 
finaxy iS Valid. Of course, the amplitude of the spectrum for most of the signals 
is greatly reduced beyond a certain frequency. That is, there is a frequency, 
fax, Up to which the power or energy of the signal sums up to at least a 95% 
of the total. To prevent the signal from being distorted by overlap between the 
remaining part of the spectrum and the following one, this remaining part can 
be cut off using anti-aliasing filters prior to sampling. For example, in ISDN 
telephony all voice frequencies above 3.4 KHz are cut off using an anti-aliasing 
filter with a cut-off frequency f, = 3.4 KHz, so that the voice signal is sampled at 
8000 samples/second. 


There is practically no ideal low-pass filter having the aforementioned properties, 
since real low-pass filters do not completely clip the frequencies, which are greater 
than the cut-off frequency, but simply reduce their amplitude significantly. Thus, 
when recovering the original signal, the frequencies of the sampled spectrum 
that did not exist in the spectrum of the original signal are not completely cut off. 
The problem is solved by selecting f, > 2 fx, So that as the difference between 
the two frequencies increases, the space between successive copies of the original 
signal spectrum increases, too, resulting in a better reconstruction of the original 
signal. 
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3. The sampling function, which is a series of delta functions spaced uniformly in 


time with a period, T,, Le., y 6(t — kT,), cannot be implemented in practice but can 
be approached by rectangular periodic pulses of very short duration. If the period 
and the duration of the pulses are T, and 1, respectively, then this pulse will track 
the original signal for a period equal to its duration, t. 


The original signal, the pulse train and the sampled signal are illustrated in Figure 1.14. 
The latter consists of a number of pulses whose amplitude is the amplitude of the original 
signal within the specified time interval. The method of this kind of sampling is called 
physical sampling and has the main disadvantage that the amplitude of the sample does not 
remain constant throughout the pulse. 

Because the output signal does not get its maximum value instantaneously, the opening 
time is divided into two parts. The first part corresponds to the time the output takes to 
obtain the maximum amplitude value and is called acquisition time, t,,. The second part 
corresponds to the time duration that the pulse tracks the original input signal and is 
called tracking time, t,,. Figure 1.15 illustrates the form of such a pulse. 

The sampling function in the case of physical sampling is described by: 


n=+0,+1,+2,... (1.8) 


tt)= 1, nT,—(t/2)<t <nT, +(t/2) 
~ |1, nT, -(t/2) St <nT,+(t/2) 


FIGURE 1.14 
Physical sampling. 
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FIGURE 1.15 
Pulse format of sampled signal. 


Equation 1.8 can be written in Fourier series as follows: 


s(t)= 9 C, exp(j2anf.t (1.9) 


n=—co 
where 


C, =(f,7) sin c(nf,t) = dsinc(nf,7) (1.10) 


Time d = 2/T, is called a duty cycle, and from Equation 1.10 it follows that C, = d > C,, > 
C,, >...., meaning that the amplitude of each harmonic component of the spectrum of the 
sampled signal s(t) decreases with the increase of the order of harmonic, and consequently, 
the same happens to the spectrum of the sampled signal, x,(f). This makes the recovery of 
the original signal in physical sampling easier than the ideal case of sampling, since the 
signal is of low harmonic content, given the absence of ideal filters. This is true because, in 
physical sampling, the low-pass filter must eliminate the amplitudes of the harmonic com- 
ponents, which are already reduced compared to ideal sampling. Thus, these amplitudes 
have values that are significantly lower in the case of physical sampling. Note that, at the 
end of the recovery process, the amplitude of the output signal will have been reduced 
compared to the original since it is multiplied with the factor d < 1. Figure 1.16 shows the 
spectra of the signals x(), s(f) and x,(). 

The signal-to-noise ratio (SNR) is used to act as a standard for the evaluation of the 
signal recovered at the receiver (i.e., whether or not it contains undesirable frequencies 
which do not exist in the original signal). In other words, this is the ratio of the average 
power of the signal to the average power of noise (i.e., the undesirable components) mea- 
sured in dB and given by the relation (Eq. 1.11) with x(f) and n() being the signal and the 
noise, respectively. 


x*(t) 


S/N (db) = 10log 1, () 


(1.11) 
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-2n/t 2n/t 


FIGURE 1.16 
Signal spectra in physical sampling. 


1.6 Sampling and Holding 


In cases where it is necessary to process the signal further in discrete or digital form, it is 
necessary that the amplitude of the samples remains constant. In these cases, the sample 
and holding method is used, in which, after the end of the sample tracking-time, the ampli- 
tude of each sample is kept constant until the beginning of the next. The shape of the 
pulses is illustrated in Figure 1.15, where time, T, — z, is called holding time. The amplitude 
of the sample is kept constant by using a memory element at the output of the sampling 
system that acts as a holding element. 

Figure 1.17 illustrates a sampling and holding device. 

The capacitance, C, of the capacitor, connected to the output, is determined by two fac- 
tors: First, the product of capacitance with the resistance of the switch in position 1 must 
be small enough for the capacitor to reach the value of the input signal, x(), in a relatively 
short time. In this way, the shortest possible holding time is achieved. Secondly, the prod- 
uct of the capacitance with the input resistance of the device connected to the output of the 
sampling and holding device must be very large. Thus, the discharge time of the capacitor 


S(t) 


ELECTRONIC SWITCH 


FIGURE 1.17 
Sampling and holding device. 
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would be large enough so that the amplitude of the signal, x,(, when the switch is in posi- 
tion 2, remains practically constant, i.e, holding is achieved. The pulse duration of the 
signal, s(f), is selected rather small. The sampling procedure, as described in the previous 
paragraphs, is defined as Pulse Amplitude Modulation (PAM), and the sampled signal is 
called PAM signal. 


1.7 Non-Accurate Reconstruction of Analog Signals 


Through Equation 1.7, we can accurately recover the original signal from the samples 
if the analog signal is of finite bandwidth and the sampling frequency satisfies the 
Nyquist criterion. In real-time processing applications, the analog signal is continu- 
ously sampled and the samples are driven in a digital system that processes them and 
generates output samples at the same rate as the samples arriving to its input. The 
digital output is converted into analog while maintaining the same time scale as the 
analog signal. 

In practice, most applications require real-time processing and reconstruction. The 
in pr 

7 


: ‘ . ’ ‘ s 
function sin c, that is defined as sin c(@) = , ranges from — co to +00, so each sample 


contributes at all the reconstruction instances. Therefore, in order to generate the analog 
signal we have to wait until all samples are available. Consequently, the reconstruction 
process proposed by the sampling theorem is inappropriate, and the method described by 
Equation 1.12 is used, leading to an inaccurate reconstruction. 


too 


x)= >) 20-7) (1.12) 


n=—co 


The function g(z) is chosen to be a function of finite duration so that at each instant, t, a 
finite number of successive samples are involved in the reconstruction, x,(f). This property 
provides a real-time reconstruction. Some typical functions used in practice are the stair- 
step, the triangular, and the finite sinc. Because of its simplicity, the stair-step reconstruc- 
tion is the most widely used. 


; 1, -1/2<7t<1/2 
Step-like : p(T) = (1.13) 
0, elsewhere 


1-|t|, -1st<1 
Triangular : g(t) = I7| : (1.14) 
0, elsewhere 
sin 2T 
-k<t<k 
Finite Sinc:(t)=4 at’ 7 (1.15) 


0, elsewhere 
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Stair-step 


Triangular 


Sa(jTsQ) 


-2Ts Ts Ts 2Ts 


FIGURE 1.18 
Step and triangular reconstruction analysis. 


In Figure 1.18, the stair-step and the triangular reconstruction are illustrated in order to 


compare the quality of these two methods. 
The functions g(t) and ®(jQ) for the step method are of the form: 


1, -1/2<7t<1/2 
T)= , OGQ)= 
97) { elsewhere §2) Q 
2 


while, for the triangular method, they are: 


2 


1-[c|, -1<7<1 5) 
T)= , P§Q)= 
o7) {; elsewhere u0) Q 


The window function S,(jT,Q) is the Fourier transform of the function sinc. The triangu- 
lar reconstruction creates weaker parasitic frequencies as it has smaller ripples than the 
step, but its central lobe is significantly different from the window function, so the distor- 


tion at the actual frequencies of the signal is greater. 


1.8 SOLVED PROBLEMS 


1.8.1 A digital signal, derived from analog signal sampling, is a linear combina- 
tion of three sinusoidal signals at frequencies: 0.15, 0.2 and 0.4 KHz. 


a. Ifthe sampling frequency is 8 KHz, find the analog frequencies that cor- 
respond to the above three digital ones. 


b. If the digital signal is reconstructed using a reconstruction period T, = 
0.1 msec, which analog frequencies will the three digital ones correspond 


to in this case? 
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SOLUTION: A digital frequency g corresponds to an analog one through the relation: 
f=0-f.. 


a. If the sampling frequency is f= 8 KHz, the digital frequencies 1, yz, ~3 
correspond to the analog frequencies f, = 1.2 KHz, f, = 1.6 KHz and f; = 
3.2 KHz. 

b. If the digital signal is reconstructed with a reconstruction period T, = 
0.1 msec corresponding to a sampling frequency of f, = 10 KHz, then the 
digital frequency gy, corresponds to f, = y,10000=1500 Hz, the digital fre- 
quency , corresponds to f, = y,-10000=2000 Hz, and the digital frequency 
gy; corresponds to f; = g3;10000=4000 Hz. 


1.8.2 Determine the minimum sampling frequency for the following band pass 
signals: 
a. x,(f) is real with non-zero X,(f) only for 9 KHz<f <12 KHz 
b. x,() is real with non-zero X,(f) only for 18 KHz<f <22 KHz 
c. xX,(f) is complex with non-zero X,(f) only for 9 KHz<f <12 KHz 


SOLUTION: 

a. For this signal, the bandwidth is B = f,—f, = 3K Hz, and f, = 4B is an integer 
multiple of B, thus the minimum sampling frequency is f, = 2B = 6Kz. 

b. For this signal, the bandwidth is B = f, —f, = 4KHz and f, = 22KHz is not an 
integer multiple of B. If we set B’ = f,/5 = 4.4, then f, is an integer multiple 
of the width of B’, thus we can sample ~x,(f) using the sampling frequency 
f, = 2B’ = 8.8KHz. 

c. For a complex bandwidth signal with a non-zero spectrum for f, < f < fy, 
the minimum sampling frequency is f, = f, =f; = 5K Hz. 


1.8.3. Explain why the data transmission rate of telephony systems for digital voice 
transmission is 64 Kbit/s. 


SOLUTION: The telephone channel has a bandwidth of 4 kHz. In order to convert 
the analog voice signal to digital, we should sample at a rate of at least 8000 
samples/sec according to the sampling theorem. Digital telephony systems use 
8-bit for each level of the analog signal (2° = 256 quantization levels). Therefore, 
the information rate of the digital signal resulting from the analog speech signal 
will be: 


8000 sample/sec x 8 bits /sample = 64 Kbit/sec 


1.8.4 Ananalog signal of the form 
x(t) = 3 cos(400zt) + 5 sin(1200zt) + 6 cos(4400zt) 


is sampled at a frequency f, = 4kHz, creating the sequence x(n) Find its math- 
ematical expression, x(n). 
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SOLUTION: Obviously t= nT, =n oop Thus, the signal x[n] will be: 


x[n] = 3 cos(4002nT,) +5 sin(12002nT,) + 6 cos(44002nT,) > 
x[n]=3 cos( +5 sn Lent +6 cos{ “A 
10 40 40 


1.8.5 Ananalog signal of the form 


x, (t) = cos(10zt) + cos(20zt) + cos(40zt) 


is sampled to produce a sequence, x(n). Calculate the sampled sequence for 
the following sampling frequencies: (a) 45 Hz, (b) 40 Hz, and (¢) 25 Hz. 


SOLUTION: x,(f) = cos(10zt)+cos(20zt)+cos(40zt) 
27mNn 4rn 8an 
a. F,=45Hz Thus, x[n]= cos e. + cos cs +cos aa 


b. F,=40Hz Thus, x[n]= cos (=) +cos (=) + cos(7n) 
c. F,=25Hz Thus, x[n]= cos( 22) + cos( 4) + cos( 52) 


1.8.6 For a continuous-time signal, we write: 


x(t) = 1+ cos(2z f,t) + cos(27 f,t), with f, > f, 


The signal is samped at a sampling frequency, f, which is 10 times the 
minimum sampling frequency. Write the expression of the sampling signal 
in the time domain. 


SOLUTION: The maximum frequency of the signal x(f) = 1 + cos(2zf;t) + cos(2zf,t) is 
fax = fo. Therefore, the minimum sampling frequency is fo min = 2fmax = 2/2 


The signal is sampled at frequency f, = 10f, min = 20f. with a sampling period 


1 
=~ =>5,_,- The expression of the sampling signal in the time domain is: 


* fs 20f, 
x,[n] = 1+ cos(2a f,nT,) + cos(2z f,nT,) 
1.8.7 A continuous-time signal is given by: 
x, (t) = 3.085 cos 1000zt — 0.085 sin 700zt — 3 sin 300zt 


where ¢ is given in seconds. Find the maximum sampling period that can be 
used to produce the perfect reconstruction of the signal x,(f) from its samples. 
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SOLUTION: Obviously the maximum frequency present in the given signal is 
Q.nax = 10002 rad/sec. According to the Nyquist criterion, the maximum sam- 
pling period would be T, 4, = #/Qmax = 1 msec. In fact, because Q,,,, is known 
(as it is the frequency of a sinusoid term in the signal), T, should be strictly less 
than 1 msec. 


1.8.8 The signal x(f) = cos(2z800t) is sampled at 600 samples/sec creating the 
sequence x[n]. Find the mathematical expression describing x(n). 


SOLUTION: The signal after sampling is given by 
x[n] = cos(2m 800nT,) = cos(2z (200 +600)nT,) = 


= cos(2z-200nT, + 22 -600nT,) = cos(2m -200nT, + 27 -600-n / 600) = 
= cos(2m -200nT, + 22n) = cos(2m -200nT,) 


We notice that the signal, at 800 Hz, has a copy at 200 Hz because it is sampled 
at 600 Hz < 2 800 = 1600 Hz. 


1.8.9 The following analog signals, x,(f) = cos(2z10#) and x,(f) = cos(2750#), that are 
sampled at a sampling frequency of 50 Hz create the sequences x,[n] and 
x,[n]. Find the mathematical expression of x,[n] and x,[n] and comment on the 
conclusions. 


SOLUTION: Sampling at f, = 50Hz: 


x,[n] = cos(2710/40n) = cos(nm/2) 
x,[n] = cos(2750/40n) = cos(5n7/2) 
But 52/2 =2/2, thus x,[n] = x,[n]. 


Consequently, x,[n] and x,[n] are identical. Therefore, frequency 50 Hz is 
folded back to frequency 10 Hz when sampled at 40 Hz. All frequencies (10 + 
40k) Hz, k = 1, 2, ... are folded back to 10 Hz, resulting in an infinite number of 
sinusoidal continuous-time functions being represented, after sampling, by the 
same discrete-time signal. 


1.8.10 For the signal x(f) with spectrum 


X(f)= coon] feu tg ] J =1/16) 


calculate the minimum sampling frequency as well as the form of the signal 
x(f) at the time-domain. 
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SOLUTION: The spectrum of the signal for f, = 1/16 is written: 


X(f= cos( Lup 4)- u(f — 4) |= 


X(f)= cos Frc £ (1) 


The second term corresponds to a square pulse with a range of 8Hz (i.e., from 
-4Hz to 4Hz) symmetrically located at the origin of the axes, therefore the spec- 
trum of X(f) is illustrated graphically in the following figure: 


From the figure, it is obvious that f,,,, = 4 Hz, therefore, the minimum sampling 
frequency will be F, = 8Hz. 
It is known that: 


cos(2z ft, yeat( £) o 5 6t —t,)+d(f+ t,) | -8sin c(8t) = 


(2) 
= 4] sinc(8(t-t,)) + sin c(8(t +#,)) | 
Thus, the signal x(f) at time domain for t, = 1/16 is given by: 
x(t) = 4| sin c(8t — 0.5) + sin c(8t + 0.5) | (3) 


1.8.11 The signal y(t)= V2 cos(4z f,t)+4cos(zf,t) is sampled at a sampling fre- 
quency that is 20 times the minimum sampling frequency. Write the expres- 
sions of the sampled signal, y,[n], in the time domain and in the frequency 
domain, Y,(f). 


SOLUTION: The maximum frequency of the signal y(t) = 2 cos(4z fit) + 4cos(z ft) 
is fnax = 2f;, So the minimum sampling frequency is f, = 2fnax = 4ft. 
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For a sampling frequency of 20 times the minimum sampling frequency, the 
sampling period is such that the expression of the sampling signal in the time 
domain is: 


yln] = y(t) = V2 cos(2n2 f,nT;) + scos( 20 fr ni) => 


t=nTs 


yl] = y(t) = eos{ }+-4eo | an (1) 


t=nTs 


The Fourier transform of the sampled signal is: 
Y(f)= Wy (5(f-2/)+5f +2f) + a(a(- a tee £)) Q) 


Therefore, the sampled signal in the frequency domain will be: 


X(f) = fs > [ X(f - mfs) |= 


m=—co 


=| | V2 (CF - ifs - 2A) + OCF - mfs +2) + 
X= 80f 2 sal s{ pm —4 J 3( ¢- mf + fy] (3) 


Replacing the sampling frequency we obtain: 


“|, V2 (5(f - m80f, -2f,)+ 6(f —m80f, +2f,))+ 
X= 80f 21/5 a 
(f) Dy 2 a(a(s- moo, fs a{f- m80 f, + fy] 


1.8.12 Suppose that we have the signal x(t) = asinc(b*#), a > 0. At which value of b 
does the signal have a sampling period of 100 msec? 


SOLUTION: The spectrum of the signal X( f) is computed next. 
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Using the Fourier transform property of time scaling we have: 


sinc(t) <> rect( f) > 


1 
sinc(B*t)< > = re 4) = Asinc(B*t) > = rect (4) @) 
B B B B 


The maximum frequency that the signal X(f) contains, equals B*/2, 
so the sampling frequency is f, = 2B?/2 = B*, and the sampling period is 


T= = = 100ms = B= 4/10. 


The spectrum of the signal X(f) is illustrated in the following figure. 


1.8.13 Consider the case of the signal x(f) = 2cos*(40zt) + 5sin(20zt). Starting from 
time t = 0, write the sequence of bits resulting from the sampling and coding 
of the first 5 samples (0, Ts, 2Ts, ..., 5Ts) of the signal. In order to compute the 
quantization levels, consider that the maximum value of the signal is 7 V and 
the minimum is —7 V. 


SOLUTION: The expression of the discrete signal x[nT,] is x[nT,] = 2cos*(40anT,) + 
5sin(20anT,) (1). The interval between samples is T, = 1/F, = 1/400 s. We calculate 
the first five samples of the signal: 


t =1T,,x[T,]=2cos?’ (408) 5in 
400 


t = 2T,,x[2T,]=2cos” Gar 5sin( 4) = 2.85 
400 0 
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t = 3T., x[3T,] =2.cos? (220) + 5sin| OO = 2.96 
400 400 


far morta aees?| °°" 1a Gain| 7 | aaa 
400 400 


In the case of 4 quantization zones, the range of each is going to be [+7-(-7)]/4 = 
3.5 V. Therefore, from —7 to -3.5 V, we will have a codeword 00; from —3.5 to 0 V we 
will have a codeword 01; from 0 to 3.5 V will have a codeword 10; and finally, from 
3.5 to 7 V would have a code word 11. Thus, the sequence 2, 2.59, 2.85, 2.96, 3.13 will 
be coded as: 10 101010 10. 


1.8.14 Explain how a voltage signal is coded between —20 V and 20 V if 3 bits are 
used. 


SOLUTION: If 3 bits are used, 8 quantization levels are required. 
The quantization step is q = (Vmax — Vmin)/2" = (20 — (-20))/8 = 5. We divide the 


max 


range into 8 levels of a height of 5 each, and at the center of each level we assign 
a value from 0 to 2"- 1. 


The analytical calculations of the aforementioned procedure are given in 
Table 1.1. 


1.8.15 A digital signal produced by the sampling of an analog signal, is a linear 
combination of three sinusoidal signals at frequencies of 0.15, 0.2 and 0.4. 
a. If the sampling frequency is 8 KHz, find which analog frequencies the 
above three digital frequencies correspond to. 
b. If the digital signal is reconstructed with a reconstruction period T, = 
0.1msec, what analog frequencies would the three digital frequencies cor- 
respond to in this case? 


TABLE 1.1 
Analytical Calculations 

Mean Value 3 Bits 
Levels of Level Encoding 
—20 to 15 -17.5 000 
-15 to 10 -12.5 001 
-10 to5 7.5 011 
—5 to0 2.5 010 
0 to +5 2.5 100 
+5 to +10 7.5 101 
+10 to +15 12.5 110 


15 to +20 17.5 11 
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SOLUTION: 


a. The frequencies of the digital signal are f, = 0.15, f, = 0.2 and f, = 04. It 
is known, that a digital frequency, f, is assigned to an analog frequency, 
fy through the relation f, = ff,. Therefore, for the sampling frequency f, = 
8KHz, frequencies f,, fo, fz, are assigned to f, = 1.2KHz, f, = 1.6KHz, f; = 
3.2KHz. 


b. If the digital signal is reconstructed using a sampling period T, = 
0.1 ms corresponding to a sampling frequency f, = 10 KHz, then fre- 
quency f, is assigned to f,,, = f,10000 = 1500Hz, frequency f, is assigned 
to fi... = f210000 = 2000Hz and frequency f; is assigned to f,, 3 = f,10000 = 
4000Hz. 


Hence, if f,, f., f; correspond to the rate of the analog signal on which the 
sampling was implemented, then the answers are the three first frequencies, 
whereas the next three values of frequencies correspond to the reconstructed 
signal. 


1.8.16 Find the Nyquist frequency for the signals: 


sinc? (100#), 0.15sinc?(100#), sinc(100#) + 3sinc? (60t), sinc(50t)sinc(100t). 


1 o 
SOLUTION: The signal sinc?(100#) i 1 to the signal —— tri] —— 
e signal sinc?(100#) is equal to the signa 100 (52. 


mum frequency of the signal is 200z rad/sec. The Nyquist frequency will be 
@, = 4002 rad/sec. 

The signal 0.15 sinc?(100t) differs only in amplitude from the signal 
0.15sinc?(100t) so the spectrum is not affected, and the Nyquist frequency will 
be w, = 400z rad/sec. 


The signal sinc(100f) +  sinc?(60f) is equivalent to the signal 


so the maxi- 


a ret ae +3 ie ri 2) so the maximum frequency of the signal is 


1202 rad/sec. The Nyquist frequency will be w, = 240z rad/sec. 
The signal — sinc(50f)sinc(100t) is equivalent to the _ signal 


5 reel 2 a ret 52 so the maximum frequency of the signal is 


1002 rad/sec. The Nyquist frequency will be w, = 200z rad/sec. 


1.8.17 Calculate the sampling frequency for the analog signal x(f) = et 2 0. 


SOLUTION: The Fourier transform of the signal x(f) is X(jQ) = es with ampli- 
For the relation X(jQ) = 0 to apply, it should be f,,,,. 00. 


tude |X(jQ)| = 


Q? 


Therefore, the sampling period must be zero, which is not possible. Let us 
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choose as Q,,,, the angular frequency that contains 95% of the signal energy. 


Then: 
Qmax co 
| xaajae-a|x(iajaas 
0 0 
OQmax és 
} x(ja|da=0.95/ 1 go 
0 » Vit Q? 
Thus, 
tan“"(Q) f= 0.95 tan“ (Q) [j= 0.95 (tan“"(e»)— tan"(0))0.95-5 2) 


We solve the relation (2): 


tan“!(Q) |S» = 0.95 5 = tan“ (Q,,,.) = 0.95.5 >2_.= tan{ 0.95 “| => 


Qurax = 12.7 rad/sec 


Therefore, the sampling frequency is: 


Q 12.7 
>2 =2) —™* |= —— = 4.04 samples/sec. 
i je ( on 1 p / 


1.8.18 Consider the continuous-time signal: x(f) = 3cos(2000zt) + 5sin(6000z#t) + 10cos 
(12000zt). 


What is the discrete-time signal that is produced if we use f, = 5000 
samples/sec, and what is the continuous-time signal resulting from the 
reconstruction from its samples? 


SOLUTION: The maximum frequency of the signal is f,,,, = 6000 Hz Therefore, 
according to the Nyquist criterion, the minimum sampling frequency that 
guarantees the correct reconstruction of the continuous-time signal from its 
samples is f, min > 2fmax = 12000 samples/sec. 

The signal is sampled with frequency f, = 10f,min = 20f,, thus with a sam- 
1 1 
pling period T,=— = 
f, 20f, 
cos(2zf,nT,) + cos(2 zf,nT,). 


. The sampled signal in the time domain is: x,[n] = 1 + 
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Sampling with sampling period T, = 1/f, = 1/5000 sec yields: 


snl = x10.) =3cos2n{ £}+5sin2n( 3} 10c0824[ © 
1 ; 2 1 
= Scos2n( 1 }n+5sin2x[1-2]n-+10c0s2e[ 1+ 1 Jn 


=3cos2n{ E)n-+Ssin2e{—2 }n+10c0s2e{ 2p 
5 5 5 
a 13cos2n( b)n—Ssin2{ 2 )n 

5 5 


That is, the frequencies contained in the interval (-f,/2,f,/2), not exceeding the 
Nyquist frequency f,/2, are f,/5,2 f,/5. 

Consequently the continuous-time signal resulting from the reconstruction is: 
x,(f) = 13 cos 2000zt-5 sin 4000zt. We can see here the results of the aliasing result- 
ing from the low sampling frequency. 


1.8.19 A continuous-time signal x,(f) consists of a linear combination of sinusoidal 
signals at the frequencies 300 Hz, 400 Hz, 1.3 KHz, 3.6 KHz and 4.3 KHz. 
The signal is sampled with a frequency of 2 kHz, and the sampled sequence 
passes through an ideal low-pass filter with a cut-off frequency of 900 Hz, 
generating a continuous-time signal y,(f). What frequencies are present in the 
reconstructed signal? 


SOLUTION: Since the signal x,(t) is sampled at a rate of 2 KHz, there will be 
multiple copies of the spectrum at the frequencies F; + 2000k where F; is the 
frequency of the ith-sinusoid component of x,(f). Thus, 


, = 300 Hz, 
FE, = 400 Hz, 
F, = 1300 Hz, 
F, = 3600 Hz, 
F, = 4300 Hz, 


F,, = 300, 1700, 2300, ... , Hz 


F,, = 400, 1600, 2400, ... , Hz 
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F,,, = 700, 1300, 3300, ... , Hz 
F,, = 400, 1600, 3600, ... ,Hz 


F;,, = 300, 2300, 4300, ... ,.Hz. 


Therefore, after filtering with a low-pass filter with a cut-off frequency of 
900 Hz, the frequencies present in y,(f) are 300 Hz, 400 Hz and 700 Hz. 


1.8.20 The following analog signals are given: 


| €[0,8]msec. siiee= -1,t €[0,8]msec. 


() cos - 
x, (j= msec 
: 0,t €[0,8]msec. 


0,t €[0,8]msec. 


a. Sample the signals so that we get nine samples in the [0,8] msec inter- 
val. (Take the first sample at 0 sec, and the ninth at 8 msec.) What is the 
sampling frequency, f, and what are the exact values of the respective 
samples? 

b. Assuming 3-bit and 4-bit rounding quantization, respectively, for level 
coding by arithmetic with two’s complement, what are the respective val- 
ues of the samples of both signals after the quantization? 


SOLUTION: 


a. The analog signal x,(f) is written: x,(f) = cos( 1) = cos zal 7 
4m sec 8msec 
for t €[0,8msec]. So the period of the signal is T = 8m sec. 


In order to have 9 samples in the interval [0,8msec], sampling frequency 


should be T, = 2 = 8msec = lmsec. Thus, f, = ae 1kHz. 
8 8 T, 1msec 


The exact values of the samples of both signals are: 


x,(0) = cos{ *-0) =1,x%,(01)= cos( 1] = (2) = cos( £2] =0 


x,(3) = cos(™-3)=-S2, ,(4)= cos( ¥-4)=-1,216)= cos( *-5]= v2 
4 2 4 4 2 


2 


1 1 : an 
x,(6) = cos( *-6] =0,x,(7)= cos( *-7] = “5 xen x,(8) = cos ri 8| =1. 
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For the second signal, we have: 


X>(0) = x, (1) = x, (2) = x, (3) = x,(4) = x,(5) = x, (6) = x,(7) = x, (8) = -1. 


b. For quantization of 3 bits level > 2° = 8 levels 


So, the two signals after the output of the 3-bit quantizer will have the fol- 
lowing values: 


%,(0) = 0.75, ¥,(1) = 0.75, %,(2) = 0, ¥,(3) = 0.75, %,(4) =-1, %,(5) =-0.75, %,(6) =0, 
£,(7) = 0.75 Kou ¥,(8) = 0.75 


whereas 


X,(0) = (1) = X,(2) = (3) = X,(4) = X,(5) = X,(6) = X,(7) = x,(8) = -1. 


For quantization of 4 bits, there are > 2' = 16 levels, and the quantization step is 


Therefore, the two signals at the output of the 4-bit quantizer will have the 
following values: %{(0)=0.875, %/(1)=0.75, %{(2)=0, *1(3)=-0.75, x{(4)=-1, 
%1(5) = 0.75, X1(6) =0, X{(7) =0.75 and x;(8) = 0.875, 

whereas 


X5(0) = 5(1) = X5 (2) = (3) = 44(4) = 245) = 23 (6) = 23 (7) = (8) = -1. 


1.8.21 Consider the signal processing system illustrated in the figure below. The 
sampling periods of the A/D and D/A converters are T = 5ms and T’ = 1 ms, 
respectively. Calculate the output y,(f) of the system if the input is x,(f) = 
3cosl00zat + 2sin250zt. The post-filter cuts off the frequencies above F,/2. 


Xait) Vatt) 


Postfilter 


SOLUTION: The sampling frequencies of the ADCs and DACs are F,; = — = 200 


samples/sec, F,, = 1000 samples/sec. 
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The signal x(n) is: 


x[n] = 3 cos me n+2sin 200n n= 3cos| | +2sin( 52") > 
200 0 2 4 


Thus, the output y,(¢) of the system is: 


V.(=3e0s{ = 10), 2sin( 91000), 


=3cos 500at —2sin750zt + ...... 


The post-filter filters the frequencies over 500 Hz. So, the output of the system 
is: 


y, (f) = 3.cos 500zt — 2 sin 750zt. 


1.8.22 The continuous-time signal x(f) = cos(1000zt) + cos(2400zt) is sampled with a 
frequency of 2 kHz. The resulting discrete-time signal, x[n], passes through 
a digital signal processor with a unit amplitude response for all frequencies. 
An ideal analog low-pass filter with a cut-off frequency of 1 KHz is used to 
reconstruct the analog signal y(f) from the output y[n] of the digital signal 
processor. 


=> Ata 


Impulse vain 
digital sequence 


x(t) 


Compute the spectrums of x(t) and of x[n]. 


SOLUTION: The signal x(t) = cos(1000z#) + cos(2400zt) is written as 


1; = 1); = 
x(t) = cu ig ee ee deg 240m 
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By using the formula X(jQ)= 2x) 4,0 (Q-Q,)k) for a periodic signal, we 
k=-co 
derive relation (2) that describes the spectrum of the signal x(£). 


X(jQ)= 2n{ 30 (Q+1000r)+5(Q— 1000) + 


vaa( 20 (Q+2400m) +5(Q- 200m) - 


X(jQ) = n(5(Q+1000r)+ 5 (Q-1000z)) + 
+n (5 (Q+ 24007) + 5 (Q-2400z)) (2) 


The signal x[n] resulting from the sampling is 


1), ae 1.3 = 
x[nT] na (ont +e eee) ae =o +e ee 
2 2 


x[n/2000] = x(cinse 4 e 05m) 4 5(e “ eae 


Lai 1 (3) 
or x[n] = (ger if e 05m) + (gr +4 eue™) 
2 Z 
So the spectrum x[n] will be 
X(e!) = (5 (w+ 0.51) + 6 (w- 0.52) + 
(4) 


+n(5 (w+ 1.22) +5 (w-1.2n)) 


1.8.23 Consider the continuous-time signal x(t) = cos(2aft) of frequency f = 400KHz. 
Sketch the sampled signal (amplitude spectrum and time-waveform) for the 
following sampling frequencies f, = f, 1.2f, 1.5f, 2f, 10f (using MATLAB). 


SOLUTION: For sampling frequency f, = f = 400KHz the code in MATLAB is: 


£f = 400e3; % f£, = £ 


s 


fs = 1.*f; 

Nt = 100; 

Ts = 1./fs; 

io 0:Ts: (Nt-1).*Ts; 


xX = cos(2.*pi.*f.*t) ; 
= (x); 

subplot (211) ,plot((0:1: (Nt-1)).*fs./Nt,abs(z)); 

subplot (212) ,plot (t,x) 
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We notice that the frequency of the reconstructed signal is f= OKHz. We repeat 
the sampling procedure for all the given frequencies, from which we reach to the 
following conclusions: 


100 
50 
) 
0 1 2 3 4 
x 10° 
2 
1 
0 1 7 1 1 
0 0.5 1 15 2 25 
x 104 


For sampling frequency f, = 1.2f = 480KHz, the reconstructed signal frequency 
is f = 81.6KHz (aliasing). 

For sampling frequency f, = 1.5f = 600KHz, the reconstructed signal frequency 
is f = 198KHz (aliasing). 

For sampling frequency f, = 2f = 800KHz the reconstructed signal frequency is 
f = 400KHz. 


For sampling frequency f, = 10f = 4000KHz the reconstructed signal frequency 
is f= 400KHz. 


1.8.24 Consider the continuous time signal x(f) = cos(7t) + cos(23t). Using three dif- 
ferent sampling periods (tsl1 = 0.05 sec, ts2 = 0.1 sec and ts3 = 0.2 sec), study 
the effect of the choice of the sampling period on the effectiveness of the 
reconstruction procedure of the signal from its samples. 


SOLUTION: The MATLAB code, with the corresponding comments on each case, is: 


tsl = 0.05; Cs2 = 0.1) tess. = 0.2; 

wsl = 2*pi/tsl; ws2 = 2*pi/ts2; ws3 = 2*pi/ts3; 

wl = 7; w2 = 23; 

t = [0:0.005:2]; 

x = cos(wl*t)+cos(w2*t) ; 

subplot (2,2,1) 

plot (t,x),grid,xlabel('Time(s)'),ylabel('Amplitude'), 
title ('ContinuousTimeSignal); x(t) = cos(7t)+cos(23t) ') 

% Sampling of the analog signal with sampling period ts = 0.05 s and 
wsl = 5.5*w2 

tl = [0:ts1:2]; 

xsl = cos(wl*t1)+cos (w2*t1) ; 

subplot (2,2,2) 
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stem(t1,xs1);grid;hold on, plot(t,x,'r:'),hold off, 
xlabel('Time(s)'), 

ylabel('Amplitude'), title('Sampled version of x(t) with ts = 
0.005s') 

% Sampling of the analog signal with sampling period s = 0.1 s kam 
ws2 = 2.7*w2 

2 = [Oste2:2] ; 

xXS2 = cos (w1*t2)+cos (w2*t2) ; 

subplot (2,2,3) 

stem(t2,xs2);grid;hold on, plot(t,x,'r:'),hold off, 
xlabel('Time(s)'), 


ylabel(‘Amplitude'), title('Sampled version of x(t) with ts = 0.1s') 

% Sampling of the analog signal with sampling period ts = 0.2 s and 
ws3 = 1.37*w2<2*w2 

t3 = [0sts3:2] > 

xS3 = cos (w1*t3)+cos (w2*t3) ; 

subplot (2,2,4) 

stem(t3,xs3);grid;hold on, plot(t,x,'r:'),hold off, 
xlabel('Time(s)'), 

ylabel('Amplitude'), title('Sampled version of x(t) with ts=0.2 s') 


or ucue Time signal; x(t)=cos(7t)+cos(23t) agmpled version of x(t) with ts=0.005s 


Amplitude 
Amplitude 


0 0.5 1 1.5 2 0 0.5 1 1.5 2 
Time(s) Time(s) 
2 ampled version of x(t) with ts=0.1s ld version of x(t) with ts=0.2s 
: 1}: : : 
8 i: BUA { : 
_ P=] . | Bi ~~ FF 4 
= = 0 ~ 
2. on og . . . . Fe; 
e |? i 2 a] Pe: ®: i 
< : a A es fale 
< 0.5 1 1.5 2 


1.8.25 Let an analog signal x,(f) = cos(10at), 0 < t <1, f=5Hz. The signal is sampled with 
the frequency f, = 10Hz(T, = 0.1s). Sketch the signal after sampling and the recon- 


sin[ x(t-nT)/T | 
m(t—-nT)/T 


structed signal resulting from the relation y(t) = » x[n] 


n=—00 
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SOLUTION: For f, = 10Hz(T, = 0.1s), the sampled signal will be x[n] = x,(nT) = 
cos(10znT) = cos(zn). 
The MATLAB code for illustrating, in figures, the signals asked is: 


T = 0.2; 

n= O<10; 

xX = cos(10*pi*n*T) ; 
stem(n,x); 

dt = 0.001; 


t = ones(11,1)* [0:dt:1]; 
n = n’*ones(1,1/dt+1); 

y = x*sinc((t-n*T)/T); 
hold on; 

plot (t/T,y,'r') 


1.5 1 


0.5 


AS : ! : ! 


NOTE: The code in MATLAB for designing the sinc function is 


x = linspace(-5,5); 
y = sinc(x); 
plot (x,y); 


1.8.26 Let the continuous-time signals: 
%) (t) = cos(6zt), X5 (t) = cos(14zt) 1 Xs (t) = cos (26zt) and y (t) =x,(t)+ x, (t) +X, (t). 


a. For sampling frequency of 1KHz, sketch the samples of the given 
signals. 

b. For sampling frequencies of 10Hz and 100Hz, sketch the samples of the 
given signals. Also, sketch the signal resulting from the reconstruction 
(based on the sampling theorem). 
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SOLUTION: 


a. The maximum frequency of the signal y(f) is 13Hz. We will sketch the 
samples of the given signals for sampling frequency 1000Hz, which is 
clearly greater than 2:13Hz. Because we chose a sampling frequency that 
exceeds the minimum limit set by Nyquist, we are able not only to recon- 
struct the original signal, but also to produce much better graphs. 


The MATLAB code follows: 


fs = 1000; % set sampling frequency 1000Hz 
ts = 1/fs; % compute sampling period 


n = 0:ts:1; %set time interval 
x1 = cos(6*pi*n) ; 
x2 = cos(14*pi*n) ; 
x3 = cos(26*pi¥*n) ; 


y = X14+xX2+x3; 

subplot (411) ,plot(n,x1) ; 
subplot (412) ,plot(n,x2) ; 
subplot (413) ,plot(n,x3) ; 


subplot (414) ,plot(n,y);title('Final signal y(t) - fs=1000Hz') 
4 
0 : ) 
0 0.2 0.4 0.6 0.8 1 
4 
9 1 1 
0 0.2 0.4 0.6 0.8 1 
1 T T T 
0 1 1 
0 0.2 0.4 0.6 0.8 1 


final signal y(t) - fs=1000Hz 


5 
0 w—eEoetes aE @—". 
5 f L \ i 


0 0.2 0.4 0.6 0.8 1 


b. If we choose as a sampling frequency 10Hz < 2:13Hz, using the appropri- 
ate code in MATLAB, we will get a result that is going to differ from the 
previous one due to the aliasing effect. 


fsl = 10; % Set sampling frequency 10Hz 
tsl = 1/fsl; % compute sampling period 
nl = 0:tsl:1; %*set time interval 

x11 = cos(6*pi*nl); 

x12 = cos(14*pi*nl1) ; 

x13 = cos(26*pi*nl) ; 

yl = x11+x12+x13; 
subplot (411) ,plot(n1,x11) ; 
subplot (412) ,plot(n1,x12) ; 
subplot (413) ,plot(n1,x13) ; 

subplot (414) ,plot(n1,y1);title('Final signal yl(t) - fs1=10Hz') 
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0 0.2 0.4 0.6 0.8 1 
1 
O; | . 

0 0.2 0.4 0.6 0.8 1 
1 
of : , 

0 0.2 0.4 0.6 0.8 1 
" final signal y1(t) - fs1=10Hz 
5 
+5 1 1 1 1 

0 0.2 0.4 0.6 0.8 1 


With the following commands we will sketch the original signal y(f) and the 
one produced from its reconstruction after setting 10Hz as the sampling frequency. 
The original signal will be reconstructed based on the sampling theorem. 


Having in mind the relation y,(f)= >» y,(nT,) sin c| f(t- nT.) | where y,(£) is 


n=—c0 


the analog reconstructed signal, and y,[nT,] is the sampled signal, we get: 


for i = 1:length(y) 

yrec(i) = sum(yl.*sinc(fs1*(n(i)-nl))); 
end 

subplot (211) ;plot(n,y) ; 
subplot (212) ;plot(n,yrec) ; 


5 


es 5 Ll L L Ll 


0 0.2 0.4 0.6 0.8 


-5 


0 0.2 0.4 0.6 0.8 
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We notice that the reconstructed signal has no resemblance to the original due 
to overlapping. The signal resulting from the reconstruction is a cosine of 3Hz, 


with an amplitude of 3. 


The spectrum of the sampled signal is formed by shifting the original spectrum 
by integer multiples of the sampling frequency and by adding up the results, that 
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is, Y,(f)= ~ > Y(f + #f,). The function Ys(f) is periodic with a period, f,. Consider 


k=-00 
an interval [-f,/2,f,/2] of Y,(f), namely [-5,5]. We will find out what spectral com- 
ponents are present in this interval, and because of the periodicity we will have 
absolutely defined Y,(f). 


Because of the component x,(t) = cos(6zt) = cos(2:pi:3-t) in the original signal, 
there will be a frequency component with a frequency of 3Hz in the sampled sig- 


nal. This component corresponds to Y,(f) = = >> Y(f +f.) for k = 0 (zero shift), 
and it is in [-5,5] interval. 5 favo 

The component x,(f) = cos(14zt) = cos(2-pi-7-t) is out, but its shift cos(2-pi-(10 —7)-t) = 
cos(2-pi-3-t) is in the interval [-5,5] and is of frequency 3Hz. The component x;(t) = 
cos(26at) = cos(2-pi-13-t) is out, but its shift cos(2-pi-(13 — 7-f) = cos(2-pi-3-t) is in the 
interval [—5,5] and is of frequency 3Hz. 


So the wrong sampling with f, = 10Hz < 2:13Hz turned the frequency signals of 
7Hz and of 13Hz to signals of 3Hz. That is the reason that the signal at the output 
is a cosine of amplitude 3 (as a result of the superposition of these three signals). 


Let us choose as sampling frequency 100Hz > 2:13Hz. 


fsl = 100; % set sampling frequency 10Hz 

tsl = 1/fsl; % compute sampling period 

nl = 0:tsl:1; set time interval 

x11 = cos(6*pi*nl1) ; 

x12 = cos(14*pi*nl) ; 

x13 = cos(26*pi*n1) ; 

yl = x11+x12+x13; 

subplot (411) ,plot (n1,x11) ; 

subplot (412) ,plot (n1,x12) ; 

subplot (413) ,plot (n1,x13) ; 

subplot (414) ,plot(n1,y1);title('Final signal yl(t) - fs1=100Hz') 

The reconstructed signal will be the same as the original: 

for i = 1:length(y) 

yrec(i) = sum(yl.*sinc(fs1*(n(i)-nl))); 

end 

subplot (211) ;plot(n,y) ; 

subplot (212) ;plot (n,yrec) ; 
5 


[o) 
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1.8.27 Sketch the quantized sinusoidal wave signal using a 3-bit quantizer (using 
MATLAB). 


SOLUTION: The following MATLAB code is used for the 3-bit quantization of a 
sinusoidal waveform (8 levels of quantization). 


cle: 

close all; 
Am = 4; 
bit = 3s 

£ = als 

fs = 30; 


t = 0:1/fs:1*pi; 

y = Am*sin(2*pi*f*t) ; 

Nsamples = length(y) ; 
quantised_out = zeros(1,Nsamples) ; 
del = 2*Am/(2*bit); 

Llow = -Am+del/2; 

Lhigh = Am-del/2; 

for i = Llow:del:Lhigh 

for j = 1:Nsamples 

if (((i-del/2) <y(j)) &&(y(j) < (itdel/2) )) 
quantised_out(j) = i; 

end 

end 

end 

stem(t,quantised_out) ; 

hold on; 

plot(t,y,'color','r'); 
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Discrete-Time Signals and Systems 


2.1 Discrete-Time Signals 


Discrete-time signals are the signals defined only for a discrete set of values of the indepen- 
dent variable of time, e.g., for all integers (Figure 2.1). In this case, the independent variable 
receives only discrete values, which are uniformly distributed. Therefore, discrete-time 
signals are described as sets of samples whose amplitudes can receive constant values. 
When each sample of a discrete-time signal is quantized (i.e., the amplitude must receive 
only a finite set of discrete values) and then coded, the final signal is referred to as a digital 
signal. The output of a digital computer is an example of a digital signal. 

A signal may be discrete by nature. One example is the closing prices of the stock market 
index. However, let us consider another category of discrete signals: those resulting after 
the sampling of continuous-time signals. Sampling is being implemented at fixed intervals 
of time and is mathematically expressed by the relation x() = [nT]. The result is a sequence 
of numbers, which is a function of the variable n. 


2.2 Basic Discrete-Time Signals 
2.2.1 Impulse Function 


The impulse function is defined as: 


_|1- for n=0 
a=] for n#0 21) 


and is illustrated in Figure 2.2. 
The shifted impulse function is, then, given by 


ani fae Bek (2.2) 


The properties of the impulse function are given in the following, Table 2.1. 
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FIGURE 2.1 
Discrete-time signal. 
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FIGURE 2.2 
Impulse function. 
TABLE 2.1 
Properties of Impulse Function 
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id Stn= x[0] whenn=0 sth) -otn N= x[N]  whenn=N 
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FIGURE 2.3 
Unit sequence function. 


2.2.2 Unit Step Function 


The unit step function is illustrated in Figure 2.3 and is defined as: 


= 1,n=0 23 
dell 0,n<0 eS) 


The shifted unit step function is given by 
Ln2k 
u[n—k]= | 7 (2.4) 


Between the functions 6[n] and u[n — k], the following relations are true: 


6[n] = u[n]—ufn— and u[n]= >; S[r]. 


m=—0o 


2.2.3 Ramp Function 


The ramp function (of unit slope) is illustrated in Figure 2.4, and it is defined as: 


Ce n for n20 
a a 0 for n<0 aa 
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FIGURE 2.4 
Ramp function. 


2.2.4 Unit Rectangular Function (Pulse Function) 


The unit rectangular function is illustrated in Figure 2.5 and it is defined as: 


1k,<n<k 
rect[n]=4 "| : a (2.6) 
O,n<k,&n>k, 


2.2.5 Exponential Function 


The unit exponential function is defined as: 


Ca", n=0 
= e Df 
sir] : ,n<O e7) 


rectangular pulse sequence 
T T 


FIGURE 2.5 
Unit rectangular function. 
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The real exponential function is illustrated in Figure 2.6, where C € R anda € R, for 0 < 
a<1and for 0 > a1, respectively. 

In complex exponential signals, the quantities A and © are generally complex numbers. 
A complex number can be written as the product of its amplitude and its phase. Thus, C = 
|Cle? and a = |ale/®. By replacing the relation of the definition of complex exponential 
signals, we obtain: 


x{n] = Ic| el? i gion) 


a 


n Pp 
ein — Ic]. 


a 


or 


x[n] = Ic| : lal” cos(Qn + 8) + j|c| : lal” sin(Qn + 6) (2.8) 


When |a| = 1, the real and imaginary parts of the signal are sinusoidal signals of dis- 
crete time. When |a| < 1, the sinusoidal signals are multiplied by an exponential number, 
which decreases. Finally, for |a| > 1, the real and imaginary parts of x[n] are sinusoidal 
sequences whose amplitude increases exponentially with n. Figure 2.7 illustrates the com- 
plex exponential sequence: 


x=exp((-0.1+ j0.3)n), for -10<n<10. 


Period of complex exponential signals of discrete time: Let N be the period of the sequence 
x[n]. Then, according to the definition: x[n] = x[n + N] = Ae®" or e/@" = ei@" « ei", which is 
obviously true when e/@” = 1. This relation is true when QN = k2z. So, 


k 
Q=2n— 
a (2.9) 


Real exponential sequence , a<1 
T T T 


Real exponential sequence , a>1 
T T T 
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FIGURE 2.6 
The real exponential function. 
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Complex exponential sequence. 


Therefore, in order for the complex exponential sequence to be periodic, there must be 
é K ' 
integers k and A, such that the circular frequency, ®, can be written as Q= 27 a When this 
is true, the period of the exponential sequence is given by: 


ON =k2n >N = r= (2.10) 


The fundamental period occurs for the smallest value of k, for which N becomes an 
integer. 


2.2.6 The Sinusoidal Sequence 
The sinusoidal sequence is defined as: 


Anl= Asing@,n, n20 (2.11) 


7 0, n<0 


and is illustrated in Figure 2.8. 
Period of discrete-time sinusoidal signals: Let N be the period of the signal x[n] = A sin@n + 9); 
then, according to the definition of periodicity: 


x[n] = x[n+ N] or Asin(Qn-+ @) = Asin[Q(n+ N)+ @] = Asin[Qn+QN + 9] > 
sin(Qn + ¢~) = sin[(Qn+ g)+QN]. 
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Sinusoidal sequence 
; B 


FIGURE 2.8 


Discrete-time sinusoidal sequence. 


Based on the periodicity of trigonometric functions, this equality is true only when QN = 


2ka. Thus: Q=27 x Therefore, in order for a sinusoidal sequence to be periodic, there 


must be integers yw and J, such that the circular frequency, , can be written as Q = 27 al 


When this is true, the fundamental period of the sinusoidal sequence is given by Equation 2.9, 


ie, N= R 2H, 
Q 
—>>L_____A 


2.3 Even and Odd Discrete-Time Signals 


The presence of even and odd symmetry in a signal is of particular interest because it reduces the 

amount of time required to process it with a computer. It is noteworthy that most signals are nei- 

ther even nor odd. Each signal, though, can be written as the sum of an even and an odd signal. 
A discrete-time signal is even when: 


x[-n] = x[n] (2.12) 


for all values of n. 

The graphs of the even signals are easily recognized by their symmetry with respect to 
the vertical axis (Figure 2.9). 

A discrete-time signal is odd when: 


—x[-n] = x[n] (2.13) 


for all values of n. 
Odd signals are symmetric with respect to the origin of the axes (Figure 2.10). For a 
discrete-time signal, an even signal x,[n] and an odd signal x,[n] can be calculated, 
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FIGURE 2.9 
An even discrete-time signal. 


FIGURE 2.10 
An odd discrete-time signal. 
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and then, if added together, can result in the original signal, x[n]. Signs x,[n] and x,[n] are 
often referred to as the even and odd components of x[n]. It is: 


x,[n] = Aden (2.14a) 
x,[n] = oes (2.14b) 


2.4 Energy and Power of a Discrete-Time Signal 


The energy of a discrete-time signal is given by: 


poo +N 
E= )'|x{n]/ =E= lim > etal (2.15) 
n=-00 n=—N 


The power of a discrete-time signal is given by: 


1 +N 7 
P= lim on et 2y Ml (2.16) 


Discrete-time signals are generally classified into two categories: 


1. Finite energy signals. These are the signals for which energy is not infinite, that 
is 0 < E < oo. In the case of discrete-time signals, the average power is given by 
= lim i and, since the numerator is finite, the average power will be 0. 
Noo 2N +1 
2. Finite power signals. These are the signals for which power is not infinite, that is 


E : 
0<P<oo. From P= im N41 it is obvious that if power gets a non-zero value, 
oo + 


then energy must be non-finite, that is, E = oo. 


However, there are also signals that do not belong to any of the aforementioned catego- 
ries. Such an example is the signal x[n] = n. Periodic signals form an important category of 
signals that have infinite energy and finite power. 
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2.5 Conversion of the Independent and Dependant Variable 


There are three types of conversion of the independent variable of a signal: 


1. Time shift: If m > 0, the signal x[n — m] is delayed by m on n axis. That is, it shifts 
right on the n-axis. The signal x[n + m] proceeds x[n], i.e., it is shifted to the left on 
the n-axis. 


2. Time reversal (time reversal mirroring): The signal x[—n] is symmetrical x[—n], with 
respect to the vertical axis. 


3. Time scaling: If a > 1, the signal x[an] is compressed in time, that is, it lasts a times 
less than x[n]. If a < 1, the signal x[an] stretches over time. 


Conversion of the dependent variable causes variations of the signal along the vertical 
axis. For example, the signal x[n] + 3 results by shifting the signal x[n] 3 units upwards, the 
signal 3x[n] results by multiplying the ordinate of the signal by 3 and the signal —x[n] is 
symmetrical to x[n] with respect to the horizontal axis. 


2.6 Discrete-Time Systems 


A system can be considered to be a physical object, which processes one or more input 
signals applied to it, resulting in the generation of new signals. A discrete-time system 
processes an input sequence, x[n], to generate an output sequence y[n]. When y[n] = 
x[n], Vn, then the system is called identical. The signal x[1] is the input of the system 
whereas the resulting signal y[n] is the output of the system. The transform that the 
system induces to the input signal can be described either with a function or not. From 
a mathematical point of view, a discrete-time system can be considered as a transform, 
T, that transforms the signal x[n] applied to the input, to a signal y[n] = T{x[n]} at the 
output (Figure 2.11). 

The system of Figure 2.11 is known as a single-input, single-output system or SISO. Most 
of the systems in the physical world have more inputs and/or outputs. A system with 
multiple inputs and a single output, e.g., an adder, is known as the MISO system (multiple 
inputs, single output). Systems with multiple inputs and multiple outputs are known as 
MIMO systems (Multiple Inputs, Multiple Outputs). 


yin]J=T [ x{n] ] 
x(n] yin] 
>} Tle] -————__—_ > 
input output 


FIGURE 2.11 
Discrete-time signal. 
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Examples of discrete time systems are: the ideal discrete time delay system y[n] = x[n — I], 


the discrete-time moving average system, y[n] = Mel +m], the discrete-time system, 
m=0 


defined by the mathematical relation y[n]= S) xt and it is called the accumulator, 


k=-00 
because in one sense, it sums all values from —co up to 1, etc. 


Various algorithms of numerical analysis are discrete-time systems. Also, processors 
that work on discrete-time data can be considered discrete-time systems. A special cat- 
egory of discrete-time systems is that of the digital systems. These are systems whose 
inputs and outputs are digital signals. A large part of the modern hardware corresponds 
to digital systems. 

In modern technology, in order to study a system, four techniques are followed: 


1. System analysis 

2. System modeling 

3. System identification 
4. Model validation 


System analysis includes the study of system inputs and outputs over time and fre- 
quency. Through this study, an attempt is made to understand the complexity of the sys- 
tem and classify it into a category. System modeling aims to “limit the description” of the 
system to the basic mathematical relations that govern its behavior. These mathematical 
relations are expressed in terms of parameters that need to be defined to respond to the 
operation of the particular system at some point in time. These parameters may be con- 
stant or time-varying. System identification is intended to determine the parameters of the 
model so that its behavior is “as close as possible” to that of the system. Finally, the quality 
of the model must be checked. Different criteria are used for this purpose. 

In practice, only special categories of systems are of interest. Each category is usually 
defined based on the number of constraints acting on transform, T: A very interesting 
category of systems offered for mathematical analysis is the linear time invariant systems 
(LTI systems). 


2.7 Categories of Discrete-Time Systems 
2.7.1 Linear Discrete Systems 


When a signal x,[n] is applied to the input of a discrete-time system, T, and it gives rise to 
the output y,[”], while the input, x,[n], gives rise to the output, y,[n], then the system, T, is 
called linear if the input, x[n] = ax,[n] + bx,[n], gives rise to the output, y[n] = ay,[n] + by,[n], 
where a, b are constant, i.e., if the superposition principle is true. The signal, x[n], is called 
linear combination of the signals x,[n] and x,[n]. 

Linear systems are those for which the properties of homogeneity and additivity are 
true, i.e., the behavior of a linear system does not change when the input is multiplied by 
a constant number. Two signals applied to the input of a linear system do not interact with 
each other (Figure 2.12). 
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FIGURE 2.12 
Homogeneity and additivity in a discrete time system. 


Ina linear system: 


e The output may depend directly on time 

e The output may not depend directly on time 

e The output may result from the time shift of the input 

e The output may result from the compression/decompression of the input 


Examples of linear systems are those defined by mathematical relations such as: y[n] = 2x[n], 
y[n] = 2n - x[n], y[n] = 2n - x[n — 1], y[n] = 3n - x[2n], y[n] = 2n - x[2n — 1], y[n] = 2n?x[n — 2], etc. 

Examples of non-linear systems are those defined by mathematical relations, such as: 
yln] = 2x°[n], y[n] = 2n?x°[n — 2], etc. 


2.7.2 Time-Invariant Discrete Systems 


A system is defined as time-invariant, when its response to an input signal does not 
change with time. In time-invariant systems if the input signal is delayed by time, ty, then 
the output will also be delayed by the same time, ty. When a signal, x[n], is applied to the 
input of the discrete-time system, T, and causes the output signal, y[n], then the system, T, 
will be called time-invariant if the input x[n — k] gives rise to the output signal, y[n — Kk], for 
each k. Time-invariant is a system whose response depends on the form of the stimulation 
and not from the time it is applied. 

A time-invariant system can be identified by the form of the input-output equation. Hence: 


e Systems where the output does not depend directly on time are time-invariant. 

e Systems that shift the input with respect to time are time-invariant. Systems where 
the input has been compressed or decompressed in time are time-varying. 

e When the relation between input-output is described by a differential equation 
with constant coefficients — that is, the coefficients do not depend directly or indi- 
rectly on time, then the system is time-invariant. 
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e¢ When the relation between input-output is described by a difference equation 
with constant coefficients — that is, the coefficients do not depend directly or indi- 
rectly on time, then the system is time-invariant. 


Examples of time-invariant systems are those defined by the mathematical relations: 
y[n] = 2x°[n — 2], y[n] = 2x?[n — 4], y[n] — y[n — 2] = 2x[n], y[n + 1] + y[n — 2] = x[n + 1] — 2x[n], ete. 

Examples of time-varying systems are those defined by the mathematical relations, such 
as: y[n] = 2n - x[n], y[n] = x[2n], y[n] = x[2n], y[n] — ny[n — 2] = 2x[n], etc. 


2.7.3 Discrete Systems with Memory 


A system is called a static or memory-less system if, for each moment, its output depends 
only on the current input value. In all other cases, the system is called dynamic or 
memory-based system. In dynamic systems, the output may depend on the current input 
value but also on past or future input values. 

LTI systems are generally memory-based systems. Memory is a fundamental prop- 
erty of the convolution sum. The only case, that a discrete-time LTI system is memory- 
less, is when it satisfies the relation h(n) = 0, for n 0. Hence, h(n) = h(0)8(n); therefore, 
this is a system that corresponds to a multiplication with a constant coefficient and 
which carries out amplification or damping of the signal. The input-output relation is: 
y(n) = h (x(n). 

For example, the system y[n] = (x?[n])? is memory-less because, at a random moment, 
n, the output depends only on the input value, x[n], at that moment. Indeed, the input- 
output equation does not contain any past or future input values. Instead, the system 
re x[n+1]+ te +x[n-1] 
only on the previous input, x[n — 1], but also on the future one, x[n + 1]. 


is a memory-based system because the output depends not 


2.7.4 Invertible Discrete Systems 


A system, T, is called invertible when, there is another system, T’, which, when connected 
in series with the system, T, provides as its output the input signal of system, T. This pro- 
cedure is implemented in applications where the effect of a system on a signal must be 
removed. Concluding, we could say that, if we know the system 7” and can connect it in 
series with the original system, we end up with an identical system. 


For example, the system described by the relation y[n] = 3 x[k] is invertible and has as 


= 
an inverse system, the system with an input-output relation, y[n] = x[n] — x[n — 1]. Also, the 


system y[n] = 6x[n] is reversible and has as an inverse system, y[n] = Sail On the other 
hand, systems y[n] = x?[n] and y[n] = 0 are not invertible. 6 


2.7.5 Casual Discrete Systems 


A system is called casual when the output at any point in time 1) depends only on the pres- 
ent or the previous values of the input; that is, it depends on x[n] for n < no. In casual sys- 
tems, the output does not depend on future input values. Generally, in the case of casual 
discrete-time systems, the output at 1 will depend on the input values at n,n — 1,n — 2, etc., 
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FIGURE 2.13 
(a) Stable system and (b) unstable system. 


and not on the input values at 1 + 1, n + 2, etc. So, the system y[n]= Yixln —k] is casual, 
u k=0 


while the system y[n] = », x[n—k] is not casual. 


All memory-less systems are casual. The reverse is not true. Indeed, for a memory-less 
system, the output at any time depends only on the present value of the input (it does not 
depend on future or past input values). 

For example, the system described by the difference equation, y[n] + y[n — 1] = x[n] - 
4x[n — 1], is a system with memory because the output depends on past input values due 
to the presence of the term 4x[n — 1]. In addition, the system is casual because the output 
does not depend on future input values. 


2.7.6 Stable Discrete Systems 


A discrete-time system is BIBO stable (Bounded Input-Bounded Output stable) when, 
for each bounded input, the output also remains bounded for all instances. Namely, if 
|x[n]| <k, then |y[n]| <m, for all n. The requirement of bounded input and output signals 
is fulfilled when the system is stable (Figure 2.13). 

For example, the system described by the relation y[n] = nx[n] is unstable because, 
although the input, x[n] = 2u[n] applied in the system, is bounded, the response of the sys- 
tem will be the signal y[n] = 2nu[n] = 2r[n], which is not bounded. 

The property of BIBO stability is expressed by the following relation: 


s= y, In(k)| < © (2.17) 


k= 


2.8 System Connections 


Complex system analysis is greatly facilitated if the system is considered to be the result 
of the connection of simpler systems. The fundamental types of system connections are in 
series, parallel, mixed connection and connection with feedback (see Figures 2.14 through 
2.16). 
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x(n] y[n] 
h1[n] * h2[n] 
FIGURE 2.14 


In series connection of discrete-time systems. 
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FIGURE 2.15 
Parallel connection of discrete-time systems. 
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FIGURE 2.16 


Mixed connection of discrete-time systems. 


2.9 Convolution 


Let an impulse sequence, 6[n], be applied to the input of a discrete-time system. The 
resulting output is called impulse response and is usually denoted by h[n]. If the impulse 
response of an LTI system is known, then the problem of deriving the response for any 
input is a purely a mathematical problem solved by the convolution theorem. 

Consider an LTI system with impulse response, h[n]. If the input, x[n], is applied to it, the 
resulting output is given by: 


y[n] = x[n]* h[n] = Dy x[k]h[n — k] (2.18) 
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TABLE 2.2 

Properties of Convolution 

Commutativity x[n] * h[n] = h[n] * x[n] 
Associativity {x[n] * h,[n]} * h,[n] = x[n] * {h,[n] * h,[n]}] 
Distributivity x[n] * {h,[n] + h,[n]} = x[n] * h[n] 
Identity 6[n] * x[n] = x[n] 


6[n — k] * x[n] = x[n — k] 


Methods of convolution estimation, allow the computation of the output of an LTT sys- 
tem when the input and impulse response of the system are known. In Table 2.2 below, the 
properties of convolution are presented. 

Regarding the Physical concept of the commutativity property, after rewriting Equation 


2.18 as y[n 1= Stk —k], it is suggested that the output signal at a certain instant 
k=—00 


occurs as a linear combination of all samples of the input signal with appropriate weight- 
ing factors. The distributivity property suggests that the output of a series of connected 
systems is independent of the sequence to which they are connected, while the distributiv- 
ity property expresses the possibility to connect systems in parallel. 

There are three techniques for calculating the convolution of two finite-length signals: 


1. Analytical, using Equation 2.18 
2. With the use of a matrix (Figure 2.17), for x[n] = {x[0], x[1]} and h[n] = {h[O], h[1], h[2], 
h[3}. 
The order of terms (i.e., the value of 1) of the signal y[n] is estimated by adding 
the corresponding indices, n, of the signals h[n] and x[n]. Each term of the sequence 
y[n] is computed by adding the diagonal terms. 


3. Chunking method: The successive steps for the calculation of y[n] are: 
e The sequences x[k] and h[k] are designed 
e The reflection of h[k] is designed, i.e., h[—-k] 


FIGURE 2.17 
Computing convolution with the use of a matrix. 
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e h[-k] is shifted to the left so that the last on the right non-zero term of h[-k] 
coincides with the first on the left non-zero term of x[k] 


e h[n—k]is shifted to the right by one step. Then the terms of h[n — k] and x[k] are 
multiplied and the partial products are summed 


4. A fourth technique, far more complex than the three aforementioned, is the graph- 
ical one. The graphical technique is the only one that can be used in the case that 
one of the two signals is of infinite length. From the definition of convolution, the 
steps for its graphical approximation are: 


e Flipping over one of the signals with respect to y-axis 

e Shifting it by n 

e¢ Multiplying this flipped, shifted signal with the other one 
e Calculating the area under this product 

e Assigning this value to x[n]* h[n] at n 


Now, x[n] and h[n] are of N and M length, correspondingly, and furthermore let 


x[n]=0 
| In| _, when n<0. Then the convolution y[n] = x[n] * h[n] is of N+ M —1 length and the 


y[0] x[0] 0 0 0 0 h{0] 
y[1] x{1] = x[0] 0 0 0 h{1] 
y[2]|=| x[2] x{1] x[0] 0 0 h[2] (2.19) 
y[3] x[3] x{2] x[1] x0] 0 A[3] 
Ly[4]} | x14] x13) x{2] = x[1] = x0] | | Af4] 


Lower triangular matrix 
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The convolution of two sequences can be written in the form of a product between a 
Toeplitz vector matrix and a vector. 


yl0] AO] 
y[1] A{1] [0] 
y[2] h{2] h[1] ALO] |} x[0] 
YS] J=| h[3] [2] hf] || x11] (2.20) 
y[4] Al4] Al3] Al2] | x12] 
yl5] A[4] A[3] 
| y[6] | A[4] 
yO] } { xo] 
y[1] x{1] x{0] AO] 
y[2] x[2] x{1] x{0] hf] 
or y[3] |= [2] x[1] x[0] A[2] (2.21) 
y[4] x{2] x[1] x[0] |} A[3] 
y[5] x[2] x{1] | A[4] 
y[6] x[2] 


The characteristic of the matrices formed by the terms of the impulse response or the 
input signal is that the elements covering the same diagonal are equal to each other and 
are, therefore, Toeplitz matrices. 

The representation of the convolution as a product of a matrix and a vector — aside from 
being an additional method for its computation — is a useful tool in the search for fast 
algorithms serving the same purpose. 


2.10 Deconvolution 


Deconvolution of the system is the calculation of the impulse response of the system when 
its response to an input signal, other than the impulse function, is known. It is: 


no)= 40 and h{n]= k=0 (2.22) 


where: 11 = 1, 2, ..., (N — 1), x[0] 4 0. 
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2.11 Correlation — Autocorrelation 


The Correlation or Cross-Correlation of the signals x[n] and y[n] is defined by the inner 
product: 


ry(O= >, xfnlyln-Q , 0S e<e (2.23) 


The independent variable, ¢, is called lag. The correlation measures the time shift between 
two signals. By comparing Relations 2.18 and 2.23, one can easily observe that convolution 
implies the reflection and shift of one of the two signals, while correlation denotes only a 


shift. It is true that: x[n]*y[-n]}_ = 2 x[m]y[m — €] = r,, (2), namely, the correlation of two 


xy 


n=l 


signals x[n] and y[n] is equal to the convolution of the signals x[n] and y[—n]. 
The correlation coefficient, p,,/(/), is the value of correlation normalized to r,,(0) and r,,(0), 
which are the maximum values of r,,(/) and r,,(/), and is defined by 


£ 
Pal O= py POSES 
xyry 


co 


where E, = DS x’[n] is the energy of x[n], and E,= by y’[n] is the energy of y[n]. 


Applications of correlation are encountered in various signal processing topics (for 
example, the search for periodicity to signals with noise and the search for delay in two 
identical signals), in digital communications, etc. Shifting a signal, x[n], to itself changes 
the degree of correlation. Knowing this change can be very useful to the calculation of the 
spectrum of the signal. 

The autocorrelation sequence is defined by: 


r= >) x{njxfn- , 0 <e<0 (2.25) 


p02 .. weet (2.26) 


But, E, = r,,(0), so Relation 2.26 becomes: 


Px.(0) = ao , 0S L<00 (2.27) 
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Cross-Correlation Properties 


Ney ()) a eal (2.28) 


Pxy(0) <1 (2.29) 


Iay(0)|S fox (Oy (0) < Oo) (2.30) 


Autocorrelation Properties 


tel) = t(D) (2.31) 


ex(€)| S Tx (0) (2.32) 


Let two signals, x(n) and y(n), of finite duration defined at 0 n < N —1domainand0n < 
N — 1 domain with N < M, respectively. 
For N =3 and M =5, correlation can be written in a matrix form as: 


Ty(—4) y[4] 0 0 

Try(-3) y[3] y[4] 0 

ty(-2) | | y[2] yl3] yl Jf 2{0] 

ty(-1) f=} yf] yf2] [3] |} of] (2.33) 
Ty(0) yl0] yf] yf2] || x12] 

Ny) 0 yf0] yf1] 

mam | Lo 0 a) 

Toeplitz matrix 


2.12 Difference Equations 


Difference equations model mathematically discrete systems, in the way that differential 
equations model analog systems. The general form of an N-order difference equation is: 


Vasln — k]= Veal ~m| (2.34) 
k=0 


m=0 


with initial conditions y[-]], y[-2], ..., y[-N] and x[-l], x[-2], ..., x[-M]. 
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The general solution of Equation 2.34 can be calculated both in n domain (by methods 
similar to the solution of a differential equation in time domain), and in z domain by 
using the z-transform. In order to solve a difference equation describing an LTI system, N 
initial conditions are required for y[n] and x[n]. For a casual system, the initial conditions 
required to calculate the first value of y[n] are considered to be equal to zero, i.e., y[-I], 
y[-2], ... and x[-I], x[-2], ... are zero. If the system is non-causal, the solution of the differ- 
ence equation includes one more term derived from the initial conditions. 

The solution of the difference equation can be calculated either by application of con- 
volution, after calculation of h[n], or directly by the difference equation, writing Equation 
2.34 in this form: 


y[n] = Sy eeyin —k]+ by . x[n—m] (2.35) 


The general difference equation in the form of Equation 2.35 is called a recursive equa- 
tion because a value of the output y[n] depends on previous values, y[n—1], y[n—2], ..., of the 
output, as well. 

In the case that the coefficients of Equation 2.34 are zero for k = 1,2, ... , N, the difference 
equation becomes: 


yl] = > Om xin — mm] (2.36) 


Equation 2.36 is non-recursive since the value of the output, y[n], is described only by the 
input values and not by any other previous values of the output. 


2.13 Discrete-Time Systems of Finite Impulse Response 


If the impulse response, h[n], of a discrete-time system is of finite duration then the system 
is called Finite Impulse Response (FIR). 
For non-recursive systems, we have: 


On nn =0,1,...,.M 
h{n] =2 by (2.37) 
0 


, elsewhere 


Thus, non-recursive discrete time systems are FIR systems. 
Consider a casual system described by the equation: 
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The system cannot be physically realized because, in order to calculate the term y[n], the 


Qa, — 
future term x[n + 1] is needed, or the impulse response would be h[n]= 4 bo a ; 
0 elsewhere 

That is, h[n] is different from zero for n = -1, and the system is non-causal. It can, 
however, be implemented using a delay system or on a computer where causality is 
not forbidden. All FIR systems are stable since the relation (2.17) is valid. If the impulse 
response, /i[n], of a discrete system is of infinite duration, then the system is called IIR 
(Infinite Impulse Response). Systems which are described using recursive equations are IIR 
systems. 


2.14 Solved Problems 


2.14.1 Consider the signal x[n]. Sketch the signals x[n — 3] and x[n + 2]. 


x[n] 


SOLUTION: The signal x[n — 3] is derived by shifting the original signal by 3 units to 
the right. The signal x[n — 3] is delayed by 3 units. The signal x[n + 2] results from 
the time shift of the original signal by 2 units to the left. The signal x[n + 2] is 
ahead of x[n] by 2 units. 


x[n-3] 
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SOLUTION: The signal x[3n] is obtained by dividing the abscissae of the original signal 
by 3. After division, only those elements corresponding to interger time inter- 
vals remain. From the original signal, only the time-terms that are multiples 
of 3 remain. While the original signal extends from n = 0 to n = 6, signal x[3n] 
extends from n = 0 ton = 2. 

The signal x[n/2] is obtained by multiplying the abscissae of the original signal 
by 2. The initial signal extends from n = 0 to n = 6, while x[n/2] extends from n = 
Oton= 12. 


x[3n] 
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(n) 


2.14.3. Let two periodic signals be x,[n] and x,[n] with fundamental periods N, and N. 
What is the condition for the signal x[n] = x,[n] + x,[n] to be periodic? What is its 
fundamental period? 


SOLUTION: Since the signals x,[n] and x,[n] are periodic, we have: 
x,[n] = x,[n + N,] = x,[n + mN,] where m: positive integer and 
x,[n] = x,[n + Nj] = x,[n + kN] where k: positive integer. 
Substituting the above equations in x[n] = x,[n] + x,[n], we obtain: 


x[n] = x,[n+ mN,]+ x,[n+kN,] (1) 
If x[n] is periodic with a fundamental period N, then 
x[n] = x[n+ N]=x,[n+N]+x,[n+N] 
Relations (1) and (2) yield: 
x,[n+mN,]+x,[n+kN,]=x,[n+N]+x,[n+N] (3) 


In order for (3) to be true, it must be: 


Ni 


k 
NN. mo the quotient of the two periods should be 
2 


mN, =kN,=N or 
a rational number. 

The fundamental period N of the signal x[n] will be the LCM (Least 
Common Multiple) of the fundamental periods N, and N, given by the rela- 


tion mN, = kN, = N, provided that k and m are prime numbers. 


6 
c) x[n]= ef" examine which are periodic and find their fundamental period. 
P P 


2.14.4 For the following signals: (a) x[n] = 4cos(zn), (b) xu]=cos'( %n) and 
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SOLUTION: 
a. x[n] = 4cos(xn) 
The angular frequency is Q = a. In order for the signal to be periodic, there 
must be intergers x and A such that Q = 27 K In this case,k=1landi=2,s0 


the signal is periodic with the fundemental period N = 2. 


b.  x[n]= cos? [En] 


Based on the trigonometric identity cos* p= TA SS8, the signal x[n] is 

1 

written: x[n]= 1 + Leos{ ¥n} i.e, it is the sum of the signals: x,[n]= 5 
1 1 

d x,[n]=—cos| —n }. 
and x,[n] 5 ( 

The signal x,[n] is written in the form x,[n]= a4 1", so it is periodic 
with period N, = 1. The signal x,[n] is also periodic because the quotient 
oft 6 is a rational number. That is N = 6. 

Q, 2/3 
jan 


c. x[n]=e 4 


The signal x[n] is in the form e®" with Q= : The signal is periodic because 


it is = =8,ie., a rational number. The fundamental period is N = 8. 


2.14.5 Calculate the energy and power of the following discrete-time signals: (a) x[n] = 
u[n], (b) y[n] = (—0.2)"u[n], and (c) z[n] = Ae". 


SOLUTION: 
a. x[n] = u[n] 
The energy of the signal x[n] = u[n] is given by: 


E= y Lxtnj/ = 3 lun = Yr = 00 
1=—00 n=0 


n=—00 n= 


The power is given by: 


+N +N 


i 1 2 : 1 2 
P= fim aaa Dy = Bim garg Dy ele = 


n=—N n=—N 


+N 
Gs iii Svein Nt) 
No~ IN +1 No~A\IN4+1) 2 


n=0 
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b. y[n] = (-0.2)"u[n] 
The energy of the signal y[n] = (—0.2)"u[n] is given by: 


pe x {nj = x 0.2)" un] -y 2 


n=—00 n=0 


“too 2n oo n 
1 1 1 25 
E — — — — = 
(| ls) 1-(1/25) 24 


Since the energy is finite the average power will be 0. 
c. z[n] = Ael2” 


The energy of the signal z[n] = Ae" is given by: 


= 2 = ‘ 2 = A 2 
= > Ix[n]/ = y |Ae@"| = A? vile! — 00 
n=—00 n=—00 n=—00 


The power of the signal is given by: 


ae a yo Fa al > acim’ = 


n=—N 


+N 
Speen <1 A> 1= fe 
Noe2N4+10 £40 Nom 2N41 


— 2N+1 
Thus, P= lim (AY 1= bm aaa" 
Neo ON +1 = N>o\ DN +1 


2.14.6 Calculate the even and odd component of the sequence x[n]. 


x[n] 


SOLUTION: Each discrete sequence can be written as a sum of an even and an odd one 
given respectively by the relations: 
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x fo] => xb] + xn} (1) and xoln] = xfn]- xn} (2). 


The sequence x[n] can be written as a sum of sequences 6. 
x[n] = 6[n + 2] + 26[n + 1] + 36[n] + d[n — 7], so x[x — n] is written: 
x[-n] = 6[—n + 2] + 26[—n + 1] + 36[—n] + 6[—n — 7]. 


Calculation of the even part: Sustituting in (1), we obtain: 


x,4[n]= 5 (5ln+ 2]+ 26[n + 1] + 36[n] + 6[n —7]+ 
+ 6[-n + 2]+ 26[-—n + 1]+ 36[—n]+ 6[-n—-7]} => 


x,[n]= 5 tI-n —7]+ 6[n+2]+26[n + 1]+ 66[1] 
+ 6[-n+ 2]+26[—n + 1]+ 6[n —7]} 


1 1 1 1 
: = 4...,0,—,0,0,0,0,—,1,3,1,—,0,0,0,0,—,0,... 
Thus: Xa[7] 5) ana aan) 5) 


Calculation of the odd part: Sustituting in (2), we obtain: 


Xo[n] = 5 (an + 2]+26[n + 1]+ 36[n] + 6[n-7]- 


6[-n + 2] — 26[—n + 1] — 36[-n] - 6[-n-7]} > 


xo[n] = stolen —7]+6[n+2]+26[n+1]- 


6[-n + 2]— 26[-n + 1]+ 6[n—-7]} 


1 1 1 1 
Thus: xol[n] = , AY 80001 AN 00 Oe + 


2.14.7 Study the casuality, memory, temporal invariance and linearity of the systems 
described by the equations (a) y[n] = 4x[n], (b) y[n] = 4x[n] + 3 and (©) y[n] = x[4n + 3]. 
SOLUTION: 
a. yl[n] = 4x[n] 
At any point in time, the output depends only on the current value of the 


input, so the system has no memory, and all systems that have no memory 
are causal. 


Time invariance: Assume that the signal x,[n] is implemented at the input 
of the system, which gives the signal y, = 4x,[n] (1) at the output. 

As the signal x,[n] is shiftet by 1), the output becomes: x,[n] = x,[n — no] 
(2), and from the input-output equation, the corresponding output would 
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be: y,[n] = 4x,[n] (3). In order for the system to be time-invariant, it must be: 
y[n] = y,[n — ny]. 

By substituting (2) in (3) we get: y,[n] = 4x,[n — ny]. And from (1), we get: 
yi[n — no] = 4x,[n — nol. 

Consequently, y,[n] = y,[n — no]. Thus, the system is time-invariant. 

Linearity: When the signal x,[n] is applied to the input of the system the 
output is y,[n] = 4x,[n]. Respectively, when the signal x,[n] is applied to the 
input of the system, the output is y,[n] = 4x,[n]. The system is linear when, 
at an input signal x,[n] = ax,[n] + bx,[n], the output is ¥) = ay,[n] + by,[n]. 
Based on input-output relation, when an input x,[1] is applied to the sys- 
tem the output is: 


Yoln] = 4x [1] = 4{ax, [1] + bx,[n]} = 


4ax,[n]+ 4bx,[n] = a{4x,[n]} + b{4x,[n]} = 4ax,[n]+ 4bx,[n] 
= a{4x,[n]} + b{4x,[n]} 


Thus, the system is linear. 


. Yn] = 4x[n] +3 


At any point in time, the output depends only on the current value of the 
input, so the system has no memory and is therefore casual. 


Time-invariance: For an input x,[n], the output is y,[n] = 4x,[n] + 3 (4). If 
x,[n] is shifted by ny, the new output is x,[n] = x,[n — no] (5), and, respec- 
tively, the output is y,[n] = 4x,[n] + 3 (6). In order for the system to be time- 
invariant, it must be: y,[n] = y,[n — no]. 

By substituting (5) in (6) we get: y,[n] = 4x,[n — nj] + 3. 
Equation (4) yields: y,[n — no] = 4x,[n — no] +3 

Hence, y,[1] = y,[n — nol. 

So, the system is time-invariant. 


Linearity: Let x,[n] > y,[n] and x,[n] — y,[n]. For the system to be linear, 
the linear combination of the two input signals should give at the out- 
put the corresponding linear combination of the two output signals, i.e.: 
ax,[n] + bx,[n] > ay,[n] + by,[n]. 

Based on the input-output relation, when the input signal ax,[n] + bx,[n] 
is applied, the output is: 


y[n] = 4x[n]+3 = 4{ax,[n] + bx,[n]}+ 3 = a[4x,[n]+ 3] + b[4x,[n]] 
# ay,[n] + by,[n] 


Thus, the system is not linear. 


. Y{n} = x[4n + 3] 


The system is not a casual. This is obvious because y[4] = x[8], i.e., the out- 
put depends on future input values. The value of the output y[n] depends 
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on other values of the input but not on the current value x[n], so the system 
has memory. 

Time-invariance: When an input signal, x,[n], causes an output sig- 
nal, y,[n], then, for this case, it is: y,[n] = x,[2n] (1). Assume that the input 
signal is shifted by my over time. This is equivalent to applying a signal 
X, = x,[n — no] (2) to the input, which will cause the signal y,[n] = x,[2n] 
(3) to output. In order for the system to be time-invariant, it must be: 
y2In] = y,[n — no]. Substituting (2) in Equation (3), we get: y, = x,[2n — nol. 
Substituting in (1), we get x,[2(n — n))], therefore, y,[n] 4 y,[n — no], thus, the 
system is a time-varying. 

Linearity: The output results from the time compression of the input sig- 
nal by a factor of 2, therefore the system is linear. 


The input x,[n] gives the output signal: 


and the input x,[n] = x,[n — n,] gives the output signal: 


(5) 


vine frst when  x,[n]<0 
2x,[n] when  x,[n]20 


The system would be linear if the input y,[n] = y,[n — no] gives the output 


_ | (-)"x,[n-1] when x,[n—n]<0 


signal: y,[n] , where Yo = X)[2n]. 


2x,[n—- Np] when x,[n-—n)]2=0 
From (4) and (5) it is: ay,[n] + by,[n] = ax,[2n] + bx,[2n] or yo[n] = Xp [2n]. 
Thus, the system is linear. 
2.14.8 Examine the stability of the systems described by the following equations: 
-1)"x[n] when x[n]<0 
ln} , pe <0 and yl] = (xf)? 


2x[n] when  x[n]= 


SOLUTION: 
a. It should be considered if |x[n]| <k for every n. 
For the case that ly[n| 7 (-1)"x{n] = \x[n]| <k. Therefore |y[n]| <k. 
For the case that x[n] > 0: |y[n]| = |2x[n]| =2|x[n]| =2k|. Therefore |y[n]| < 2k. 
The two inequalities indicate that |y[n]| < 2k for all n, thus, the system 
is stable. 


b. Based on |x(#)| <k and raising to the square: |x(f)|? <k? so |y()| < , thus, 
the system is stable. 
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2.14.9 Let h[n] the impulse respons of a LT] system. Find the response, y[n], when x[n] 
is the input signal. 


x[n] 


SOLUTION: We shall compute the convolution with three different ways: 
a. Solution using the analytical technique: 
From the figure, we have: 


x[n] = 6[n] + 26[n- 1] (1) 
h[{n] = 26[n] + 6[n - 1] + 6[n- 2] + 6[n-3] (2) 
Thus, 
x[n] * h[n] = {5[n] + 26[n — 1]} * h[n] 
= O[n] *h[n]+ 26[n- 1] *h[n] 
=h[n]+2h[n—-1] (3) 
But 


h{n] = 26[n] + 6[n—1]+ 6[n- 2]+ d[n- 3] (4) 


2h[n — 1] = 46[n — 1] + 26[n — 2] + 26[n — 3]+ 26[n— 4] (5) 
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So the convolution is computed as follows: 


(3) => x[n] *h[n] = h[n]+ 2h[n- y= 
© 6) 
= 26[n]+ 56[n—1]+ 36[n —2]+ 36[n—-3]+ 26[n— 4] 


b. Solution with the use of matrix: 


We create the matrix below. Each term of the sequence y[n] is calculated by 
adding the diagonal terms. 


y[O]J=2y[aj=5 


yes YRIS 


So the signal y[n] that is produced from the convolution x[n]* h[n] is: 


y[n}-x[n}*h[n] 


c. Solution with the use of the chunking method: 
The successive steps for computing y[n] are: 
e The sequences x[k] and h[k] are designed 
e The reflection of h[k] is designed, i.e., h[—k] 


e h[-k] is shifted to the left so that the last on the right non-zero term of 
h[-k] coincides with the first on the left non-zero term of x[K]. 


{= 
e h[n — k] is shifted to the right by one step. Then the terms /i[n — k] and 
x[k] are multiplied, and the partial products are summed. 
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For n = 0: h[0-k] al 


yf] = xtK] + ht — R= 1 @ ol | 
=014+01+01+4+1.2=2 


For n =1: 


yf] = x[k] - h[—k] = 
=014+014+114+22=5 


For n = 2: 


yl2] = xIk] - h[2 — k] = 
=014+11+214+02=3 


For n = 3: 


y[3] = x1A] - h[3 — k] 
=114+21+014+02=3 


For n = 4: 


yl4] = xIK] - h[4 — kK] 
=104+21+01+01+02=2 
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2.14.10 Let a linear, time-invariant system with the following impulse response be: 


n= 1 O<n<4 
0, elsewhere 


Compute the output y[] of the system when the input signal is: 


a", 3<ns5 
x[n]= 
0, elsewhere 
N=l AN -1 
where @ is a non-zero coefficient. It is given that > A" = for) #1. 
n=0 A — 1 
SOLUTION: It is y[n] =0,n<-3,n>9. 
For -3 <n <1: 
[n]= >. [k]= s, a =a Sak = bile 
Yun = k=-3 ~ k=-3 - k=0 ~ a-l 
For2<n<5: 
n aaa 1 
= k _pyn-4 k _ .n-4 a 
y[n] >t ] “a eal o a-l1 
Finally, for 6 <n < 9: 
[n]= > x[k]= a" ak = a4 avtal 
ee k=n-4 7 k0 oT 


2.14.11 Compute the convolution, y[n] = x[n]* v[n] when 
x[n] = u[n] -—u[n—4], v[n] = 0.5" u[n] 
SOLUTION: The convolution of the two sequences is computed as follows: 


yl] = x{n]* ofr] = » x{kJo[n— k] = » (u[k] —ufk — 4])0.5""ufn —K] 


k=-00 k=-00 
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In order to graphically calculate the convolution, v[—k] is sketched and then 
shifted by n. The last right non-zero element of v[n — k] has 1 as its abscissa. 


Consider the following time-intervals: 
When n < 0, there is no overlap, so y[n] = 0. 


When 0 <n <3: For k < 0, it is x[k] = 0, so the lower limit of the sum- 
mation of the convolution will be 0. The dashed line shows the overlap 
of the two sequences that are from k = 0 to k =n. The upper limit of the 
summation will be n. 


Thus, it will be: x[n]*v[n—k]= ye = 05" 0.5" (2) 


k=0 k=0 
But, Yo. oie -Yoo 5) > is the sum of (n + 1) first terms of geo- 


k=0 
metrical pieteccion: with the first term being the 1 and ratio 2, so 


yln] = x{n]*o[n—k]= > 05" = as") 2 =2-0.5" (3) 
k=0 k=0 
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When n > 4: The dashed line shows the overlap of the two sequences, 


which is from k = 0 to k = 3. These are the limits of summation. 


3 
[n]* o[n-K]= }°0.5"* = ast Dios" = 05" 7 20 = 05405" 


0 for n<0 
Thus, y[n]= 2—0.5” for n=0,1,2,3 
0.5"*-0.5 for n24 


2.14.12 Find the casual system that gives the response y[n] = {1, 2, 3, 4, 5, 4, 2} when the 
input signal is x[n] = {1, 1, 1, 2, 2} 


SOLUTION: From Relation 2.22, we have: 


x[0] 
x[0] 1 
h[2] = (y[2] — h[0]x[2] — h{1]x[1)) 
x[0 
n[3] = thet 1-1-1 -0 
yay 2212 2 14-01 


Thus, h[n]= {h if 1,0, 0}. 


(4) 
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2.14.13 Consider the system that is characterized by the impulse response h[n] = (1/2)", 
n > 0. Find the input signal x[n] that creates the output y[n] = {0, 1, 4, 2, 0, O}. 


SOLUTION: It is: y[0] = x[0]h[0], y[1] = x[1]JA[0] + x[0]h[1], etc. 


Thus, 
_ YlO]_O_ 
x[0]= ho] 1 0 
ees 
sry = (W/LlI= x10] HEAD 2 a4 
h{0] 1 
x(2) = (y[2] — (x[0] h[2]+ x[1] h[1])) o 4—(0-(0.5)? +1-0.5) =35 
h{0] 1 


So: xl] ={0, 1, 3.5,...) 


2.14.14 Compute the first six samples of the impulse responses of the following differ- 
ence equations using the iterative method. 


i. y[n] = x[n] + 2x[n — 1] + 3x[n — 2] — 4x[n — 3] + 5x[n — 4] 
ii, y[n] = 4x[n] — 0.5y[n — 1] 


SOLUTION: 
i. For the non-recursive difference equation 


y[n] = x[n]+ 2 x[n—-1] +3 x[n—-2] -4 x[n—-3]+5 x[n—- 4] 


we create the following Table 2.3 of values 
Then, 


y[n]={.. 0,.., 0,1,2,3, —4, 5, 0,..,0,...} 


TABLE 2.3 


Values for the Non-Recursive Difference Equation 


N x[n] x[n-1] x[n-2] x[n-3] x[n-4] yin] 
0 1 0 0 0 0 1 
iL 0 1 0 0 0 2 
2 0 0 1 0 0 3 
3 0 0 0 1 0 -4 
4 0 0) 0 0) 1 5 
5 0 0 0 0 0 
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TABLE 2.4 


Values for the Recursive Difference Equation 


n x[n] y[n-1] yin] 
0 1 0 4 
1 0 4 2 
2 0 2 1 
3 0 1 -0.5 
4 0 0.5 0.25 
5 0 


ii. For the recursive difference equation 
y[n] = 4 x[n] -0.5 y[n-1] 


we create the following Table 2.4 of values 
It is: 


y[n]={..,0,.., 0,4, —2, 1, -0.5, 0.25, —0.125,...) 


3 e 


[| | | 
LLL | 
uJ . 


2.14.15 For the system of the following figure, it is given that h,[n] = u[n] — u[n — 2] 
and also that its total discrete impulse response is considered known. Calculate: 
(a) h,[n] and (b) the response of the system for the input x[n] = 6[n] — 6[n — 1]. 


———+ bitin) =—$- ss bran) © | bran) 32 XV 
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h[n] 


11 
10 


SOLUTION: Two different approaches for the solution of both parts of the problem are 
given next. 


a. It is h[n]=h,[n}* h[nF h[n] (1) 


= hs[n] ={1,2, 1, of 


(1) = h[n] = h[n]* h[n] = h{n]* (n] + 26[n - 1] + 6[n - 2]) > A[n] = h,[n] +2 h,[n-1]+ [1-2] 
(3) 
(3) = h[0] =h,[0]+ 2 h,[-1]+ h,[-2] > h[0]=1 


h{1]= h,[1]+2 h[0]+ ,[-1] > hf] = h[1] — 2n[0] = 3 
h[2] = 10 = h,[2]+2 h,[1]+ h,[0] = h,[2]=3 
h[3] = 11 = h,[3]+2 h,[2]+h,[1] > h,[3]=2 


h[4] = 8 = h,[4]+ 2 1,[3]+ 4,[2] => 1,[4]=1 
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h[n]=0 Vn25 


Second proposed solution: 
It is h[n] = h,[n]}* h,[n}* h, [1] 


(1) = h[n] = hy[n] * [7]. 


h,[n] will be computed using deconvolution. 


h(] = h,O\3[1] _ 5-1-2 _ 


3 
h;[0] 1 


h{] = hy [0] [1] + h,[1]h5 [0] = hy] = 


h[2]= SY mlevel2 ~k} = h,[0}h5 [2] + h, 141] + h, [2], [0] = 
k=0 


10=1-14+3-2+h[2]-1=> 4[2]=10-7=3 
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A[3] = h,[0}h, [3] + 1, [15 [2] + 2,[2]h5 [1] + 1,13]; (0) = 
11=1-0+3-14+3-2+h,[3]-1> 4[3]=11-9=2 


A[4] = h,[0]5[4] + h, [1513] + h,[2]5[2] + by [3]4, [1] + 
h[4]j[0] > 8=3-14+2-2+h|4]-1> 4[4]=8-7=1 


A[5] = h,[0}h3[5] + hy [1] [4] + 24[2]5[3] + 2, [3]5[2] + 
nA (114 h,[5]4[0] > 4=2-141-2+h,[5]-1> h[5]=4-4=0 


Thus, h,[n]= {1 as 2,11 


b. First proposed solution 


y[n] = x[n]* h[n] = (6[n] — 6[n — 1))* h[n] = h[n]- h[n- 1] (5) 


It is true that, 
The system response will be: 
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Second proposed solution 
It is true that y[n] = x[n]* h[n] 
x[n] = 6[n] — 6[n - 1] = {1-1} and h[n]= {1, 5,10,11,8,4, i}. 


Hence, 
y[0] = x[0]h[0] =1x1=1 
y[1] = x[OJA[1] + x[1]h[0] =5-1=4 
yl2] = x[O]h[2] + x[1Jh[1]+ x[2]h[0]=1x10-1x5=5 
y[3] = x[0Jh[3] + h[2]=11-10=1 
y[4] = x[0]h[4] + x[1]h[3] = 8-11=-3 
y[5] = x[0]h[5] + x[1]h[4] = 4-8 =—-4 
y[6] = x[O]h[6] + x[1]A[5] = 1-4=-3 
y[7] = x[0]h[7] + x[1]h[6] =0-1=-1 


So, the response of the system will be: 


y[n] = {1 4,5, 1,-3,-4,-3, 1 


2.14.16 Solve the following difference equation. 


yln] - ay[n- 1] =z[n], y[-1]=b, z[n]=cé[n] 


SOLUTION: An iterative solution of the difference equation will be given. 
For n > 0, it holds: 
y[0] = cd[0]+ ay[-1] = c + ab 
y(1) = cd[1] + ay[0] = a(c + ab) 
y[2] = cd[2]+ ay[1] = a? (c + ab) 


y[n] = a"(c+ab)=a™'b+a"c 
For n < 0, it is true that: 


yln—1]="(ylal—x0ml),— yl-t]=b 
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1 b 
2)= 1 I= 
y[-2] a wl ]—-x{[-1]) - 
1 b 
yl-3] = —(yl-2]-x[-2)) = 
a a 
b 1-n n+1 
y[-n]=—= =b-a "> y[nl=b-a 
a 
Thus, 
ae bea™t+c-a", n2=0 
u bea™, n<0 


2.14.17 Consider the following two signals of finite length: 


x, [1] = (2) (ota u[n—4]) 


~frl=(1 "(ut uln—5]) 


Compute the convolution and correlation of the two signals using a matrix-vector 
product. 


n 
4 


SOLUTION: ~,[”]= ( )- (erg u[n 4]) => nfl= 4.04, 


sted (1 "(ot uln—5]) => nfl = {041 


Computation of convolution: 


Vk ky 


k, =max(0,n—3) 
where } k, =min(4,n-1) ¢ andl <n<7 
ki sk 
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Substituting the values yields, 


Oat AO= 3 11 11 8 11 1 on 


" 4’ 10’ 10’ 10" 2’ 5’ 20’ 


2 
2.14.18 Compute the autocorrelation of the signal: x[n]= cos( 22") 


n=Ny 
SOLUTION: It is _-(!) == SY lnd-xfn-0 


n=N_, 


Here, we have: N,; = 0, Ny,=N-1 


4 51] og 2t=2at0—) cos{ 2+ 2nt— | 
N42 N - 


n=0 


1 >| (22!) & aa!) 
= cos cos = 
2N 44 NN 


10 2 1 > & >) 
= ) cos——-——— ) cos} ——-—— |=> 
2Na "NN 2NS N N 


Consequently, the autocorrelation of the signal x[n] = cos( 22") is: 


Adie 1 ee 2a = 1 “foe 2a 
2N N 2 N 


2.14.19 Consider the systems with the following input-output relations: 


x] "|, n: even (21) 
System 1(S)): y[nJ=} +? 


0, n: odd (21+1) 


System 2 ($2): yl] = x[n]+ ¥ x[n—1]+ 4 x[n-2] 
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System 3 (S3): y[n] = x[2n] 
S1, $2, S3 are connected in the way illustrated in the figure below: 


x[n [n] 
4 si |_| so | eo 


w[n] z[n] 


Find the input-output relation of this system, i.e., the difference equation. 


SOLUTION: The outputs of the three systems are given respectively by the relations: 
$3: y[n] = x[2n] 


$2: z[n] = w[n]+ soln 1]J+ ; w[n— 2] 
x "| n=2k 

Sl: w[n] = 
0,n=2k+1 


From the above relations, we compute the output: 
[n]= [2n]=wl2n]++wl2n-1]+—>w[2(n-1) ]= 
satiate 2 4 
1 1 
= x[n]+0+ qila- 1] = x[n]+ quln- 1] 


Therefore, the difference equation is: 


yn] = x{n]+ oxln -1] 


2.14.20 Calculate the impulse response of the discrete-time system illustrated in the figure 


below when: /,[n]=6[n]+ zon -1]), h[nJ= ail - Fan —1], h[n] = 28[n], 


P 2 
h,[n] = (3) u[n]. 
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-e | h1[n] ap “ 


h2[n] 


h3[n] -——@ 


h4[n] 


SOLUTION: Based on the figure, the systems with impulse responses /,[n] and h,[n] 
are connected in parallel and in series with ,[n]. The total response of the sys- 
tem is: 


h[n] = h,[n] + hl] *| hy[n] + hylr] | = byl] + h,[n] * h,[n] + h, [1] * hl] 


But, 
1 1 1 1 
fbn haln={ 2 stn) otn—1)} +2000) = 4 otn)261m—1 511) +26 J=9[n]-, oln-1] 
1 i) 
halal ={ 3 te o[n u}| (2) aos} 


Thus, h[n]= {ot + 5 an - uh + {ot - san - a} + {-d[n]} = 6[n] 


2.14.21 a. Using MATLAB, sketch the following sequences: delta[n], u[n], a", reer, r[n], 
and x[n] = 2delta[n + 2] delta[n] + e"(u[n + 1] — u[n — 2]). 


b. Sketch the odd and even part of the step sequence from —20 to 20. 
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SOLUTION: a.1. Code for designing delta[n] = 5[n] 
The easiest way is the use of gauspuls function: 
n==3°73 


d=gauspuls(n) ; 
stem(n,d) 


+ + + + ) 
-3 -2 -1 0 1 2 3 


Another way to work is to set separate time intervals and place them in 
apposition. Then we define the values of the function 6[n] in each timeframe 
and plot it. 


Hil=-3.3= 12 

n2=0; 

n3=1:3; 

n=[n1 n2 n3] 
dl=zeros(size(nl))j; 
d2=1; 
d3=zeros(size(n3)); 
d=[d1 d2 d3] 


stem(n,d) 
1 
0.8+ 4 
0.6+ 4 
0.4- 4 
0.2 4 
o¢ é 6 6 é o 
3 2 1 ) 1 2 3 


Alternatively, we can write the following code in which the sequence 6[1] 
is plotted over the time interval —5 < n < 10. The command x = (k == 0) gives 
result 1 when k is 0 and gives result 0 when k is other than 0. 


k1=-5; 
k2=10; 
k=k1:k2; 
x= (k==0) ; 
stem(k,x) 
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1 
0.8/5 7 
0.6 7 
04, 7 
0.27 =| 
0¢——»—_»—__» o—__»__» __» __¢ __» __» __» —__» 6 
5 0 5 10 


We can also create a function (impseq.m), which implements the unit 
impulse function 6[n] 


function [x,n]= impseq(n0,n1,n2) 
if ((nO<n1) | (nO>n2) | (n1sn2))\ 
error('arguments must satisfy nl <= nO <= n2') 


end 
n = [nl:n2]; 
x = [(n-n0) == 0]; 


After saving the function to an M-File, we run it from the command win- 
dow and design for example the 6(n-1). 


[x,n]= impseq(1,-3,4); 
stem(n,x) 


2. Code for designing u[n]for -3 <n <5. 


n2=0:5; 

n=[n1 n2]; 
ul=zeros(size(nl1)); 
u2=ones (size(n2)); 
u=[ul u2]; 
stem(n,u) 


1 
0.8 
0.6 
0.47 
0.27 
Oc + 
e -1 0 1 2 3 a 5 


rot) 
a 
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We can also create another function (stepseq.m), which implements the unit 
step sequence u[n]. 


function [x,n] = stepseq(n0,n1,n2) 

if ((nO<n1) | (nOsn2) | (n1sn2)) 
error('arguments must satisfy nl <= nO <= n2') 
end 


n = [nl:n2]; 
x = [(n-n0) >= 0]; 


After saving the function stepseq.m to an M-File, we run it from the com- 
mand window and design u(n — 1) for —3 < 5. 


[x,n]=stepseq(1,-3,5); 
stem(n,x); 


3. Code for designing a” yia a = 0.8 < 1 


al=0.8; 
xl=al.“*n; 
stem(n,x1) ; 


2 
1.5 | 
4 | 
0.5 | | | ; 
|_| 
3 2 4 0 1 2 3 4 5 
Code for designing a" fora =1.2>1 
a2=1.2; 
X2=a2.“n; 


stem(n,x2); 


2.5 
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4. Code for design r” el”. 


ne=10<+103 

r=0..9'3 

w=l1; 

x=(r.*n) .*exp(j*w*n) ; 

stem(n,real(x)); % plot of the real part of rei" 


2 


stem(n,imag(x)) % plot of the imaginary part of rei 
stem(n,abs(x)) % plot of the magnitude of re” 
stem(n,angle(x)) % plot of the phase of mei” 


5. Code for designing r[n] 


N=-3:6; 

r=0.5*n.* (Sign(n)+1); 
stem(n,r) 

grid 


Alternatively, we can create the function uramp.m and apply it accordingly 


function [sig,time]= uramp(n) 
time=n; 

sig=zeros(size(n)); 
sigl=find(n>=0) ; 

sig(sigl) =n(sigl)-n(sigl(1)); 
stem(time, sig) ; 

grid; 
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6. Code for designing: x[n] = 2delta[n + 2] — delta[n] + e"(u[n + 1]) —u[n — 2]) 
n = -3:3; % define time interval 
% Define x[n] 
xX e209 (nh S=.-42) = (Ch se ST) sis % 2delta[n+2] 
2 eC Se .0)) es Ch SST) eos % -delta[n] 
+ exp(n).*((n >= -1) - (n >= 2)); % e*n(u[n+1] -u[n-2] ) 
stem(n,x); % Graph x[n] with respect ton 
xlabel('n'); % label on the x-axis 
ylabel('x[n]'); % label on the y-axis 


title('x[n] = 2delta[n+2] - delta[n]+ e*n(u[n+1]-u[n-2])'); 


xin = 2delta[n+2] - delta[n]+ e"(u[n+1]-u[n-2]) 


2r J 
= 
x 
a; J 
Oe & & © 
-3 -2 -1 0 1 2 3 
n 
b. We shall design the sequences x.{u] = $(x] + x1-n)) and 
1 
xo[n]= 5 (x4 -x-n]}) 
%$ partition odd - even 
n=-20:20; 


m=length(n) ; 

k=find (n==0) ; 

u=zeros(1,m); 

u(k:m)=1; 

ur=u(m:-1:1); 

% even part xe(n)=1/2[x(n)+x(-n)] 
ue=1/2* (utur) ; 

% odd part xo(n)=1/2[x(n) -x(-n) ] 
uo=1/2* (u-ur) ; 

% graph of signal x, xl ,x2 
subplot (3,1,1); 

stem(n,u); 
title('original'); 
axis([-20 20 -1 1.5]) 
subplot (3,1,2); 

stem(n,ue) ; 

title('even part') ; 
axis([-20 20 -1 1.5]) 
subplot (3,1,3); 

stem(n,uo); 

title('odd part'); 
axis([-20 20 -1 1.5]) 
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original 
1 csbcnsesessesesess LIUMIUIIIIIIM INT) 
0. 48 0S 0 5 10 15 20 
even part 
‘ pevevececoecorcoeceeleceeoeceececcecores 
to. 45. 40.5 0 5 10. 15-20 
odd part 
1 I I I I I I I 4 
Ofsssapasssgsasagasasct tttererererere rey 
20 15 -10. -5 0 5 10 15 20 


2.14.22 Let the sequence x[n] = 0.9" for -10 < n < 10 and x[n] = 0, elsewhere. Design using 
MATLAB the sequences x[n —10], x[n + 10], x[2n], x[n/2] and x[—n]. 


SOLUTION: The code in MATLAB for designing the signal x[n] = 0.9" 


n=-20720; 

m=length (n) ; 

k=find(n==0) ; 

a=0.9; 

x=zeros(1,m); 

x(k-10:k+10) = a.*n(k-10:k+10); 
stem(n,x); 


Nh 
T 
i 


= 
T 


Definition and design of the signal x[n —10]: 


x1l=zeros(1,m); 
x1 (k:m) =x(k-10: k+10); 
stem(n,x1); 
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= 
T 


Definition and design of the signal x[n + 10]: 


x2=zeros(1,m); 
x2 (k-20:k) =x (k-10:k+10) ; 
stem(n,x2); 


| Wt. een 


-20 -15 5 10 15 20 


Definition and design of the signal x[2n]: 


x3 =zeros(1,m); 
x3 (kK-5:k+5) =x(k-10:2:k+410); 
stem(n,x3) 


= 
T 


tne 


-20 “15 -10 5 10 15 20 


Definition and design of the signal x[n/2]: 


x4=zeros(1,m); 
x4(1:2:m)=x(k-10: k+10); 
stem(n,x4); 
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Definition and design of the signal x[—n]: 


x5 (1 


:m)=x(m:-1:1 ); 


stem(n,x5) 


Ds) 
T 


= 
T 


eee eet III pened 


5 10 15 20 


2.14.23 a. Write a function that implements a shift to a discrete-time signal. Implement 
and design the signal y[n] = x[n—2] for x[n] = [1, 2,3, 4],0 <n <3. 

b. Write functions that implement frequency division and frequency multiplica- 
tion, respectively, to a discrete-time signal. 


SOLUTION: 


a. We create the function sigshift.m that implements a shift to a discrete signal 


oe 


2 
oO 


n 


function [y,n] 


= sigshift (x,m,n0) 
y(n) = x(n-n0) 
x is the signal, 
= m+n0; y = xX; 


mis the time , nO the shit 


In order to implement and design the signal y[n] = x[n—2] for x[n] = [1, 2, 3, 4], 
0 <n <3, we write the code: 
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[y,n]=sigshift (x,m,n0) 
stem(n,y) 


b. We create the function sigfreqdiv.m that implements the frequency division to a 
discrete-time signal. 


function y=sigfreqdiv (a,x) 
nl=length (x) ; 
m=floor(nl/a) ; 
for i=1:m 

y (i) =x (i*a) ; 
end; 


We create the function sigfreqmul.m that implements the frequency multi- 
plication to a discrete-time signal. 


function [y] =sigfreqmul (a,x) 
nl=length (x) ; 
m=nl*a; 
for i=l1:m 
y(i)=0; 
if mod(i, a)==0 
y (i) =x(i/a) ; 
end; 
end; 


2.14.24 Write a code in MATLAB to compute the linear convolution of the functions: 
x{m]=h{n]=[123 45 6]. 


SOLUTION: The code is: 


cle; 

clear all; 

close; 

disp('enter the length of the first sequence m='); 

m=input (''); 

disp('enter the length of first sequence x[m]='); 

for i=1:m 
x(i)=input(''); 


end 
disp('enter the length of the second sequence n=') ; 
n=input(''); 


disp('enter the length of second sequence h[n]='); 
for j=l:n 

h(j)=input (''); 
end 
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y=conv(x,h) ; 


figure; 

subplot (3,1,1); 

stem (x) ; 

ylabel ('amplitude---->'); 
xlabel ('n---->'); 


title('x(n) Vs n'); 
subplot (3,1,2); 


stem(h) ; 
ylabel ('amplitude---->'); 
xlabel ('n---->'); 


title('h(n) Vs n'); 
subplot (3,1,3); 


stem(y) ; 
ylabel ('amplitude---->'); 
xlabel ('n---->'); 


title('y(n) Vs n'); 
disp(The linear convolution of x[m] and h[n] is y'); 


So, the linear convolution of x[m] and h[n] is: 
y = 14 10 20 35 56 70 76 73 60 36 


2.14.25 Design the signal: 


x[n] = 2delta[n+ 2] — delta [7] + e% n(uln +1]-u[n- 2]) 


SOLUTION: We write the code: 


oe 


signal x[n] = 2delta[n+2] - delta[n] 

+ e*n(u[n+1]-u[n-2]). 
= -3:3; % define discrete-time variable 
define x[n] 


oe 


5 


oe 


x = 2.*((n >= -2) - (n >= -1)) % 2delta[n+2] 

- 1.*((n >= 0) - (n >= 1)) % -delta[n] 

+ exp(n).*((n >= -1) - (nm s= 2)); % e*n(u[n+1]-u[n-2]) 
stem(n,x); % plot x[n] vs n 
xlabel('n'); % label plot 


ylabel('x[n]'); 
title('A Combination Signal') ; 


A Combination Signal 
T T 
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2.14.26 A basic discrete-time signal that appears quite often is: x[n] = A cos(w gn + 9). 
Write a function that produces a sinusoidal signal of finite-length and receives 
five input values: three for the signal parameters and two for determining the 
first and the last signal index of n. The function will return as the output, a 
column vector containing n values and a column vector containing the corre- 
sponding signal values. Then, using this function, design the signal 2sin(zn/11) 
for —20 <n < 20. 


SOLUTION: The code for the function is: 


function [x,n]=sinusoid(A,w,phi,n1,n2) 

% SINUSOID Creates a discrete time signal 

% The command [x,n]=sinusoid(A,w,phi,n1,n2) 

% creates the signal A*cos(w*n+phi) at the time interval 
% ni<=n<=n2. 

n=[n1l:n2]'; 

x=A*cos (w*n+phi) ; 


Now in order to create the signal 2 sin(zn/11) for 20 < n < 20, we run the 
function as follows: 


>>[x,n]=sinusoid(2,pi/11,pi/2,-20,20) ; 
% Plot of the signal 

>> stem(n,x) 

>> grid 

>> xlabel('n') 

>> ylabel('sin(pi n/11)') 


4. Plot the output of the 


2.14.27 Consider the input sequence x[n] = [1, 2,3,4,5],0<n< 
,2,1],-1<n<l. 


system described by the impulse response h[n] = [ 
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SOLUTION: 
First suggested solution: 


We define the two signals at the same time interval. So, we have to compute 
the convolution of the input signal x[n] = [1, 2, 3, 4, 5],0 <n < 4 and the impulse 
response hi[n] = [1, 2, 1,0, 0,0], -l1<n <4. 


We plot the two signals: 


n=-1:4; 

x= [0.,1,2), 344751 4 
Hea [13.25.1050 ; 0] % 
subplot (121); 
stem(n,x); 
axis([-1.1 4.1 -.1 5.1]); 
legend('x[n] ') 

subplot (122) ; 
stem(n,h) ; 
axis([-1.1 4.1 -.1 5.1]); 


legend('h[n] ') 

5F 5F : 1 
—* x(n] —* hf] 

4 4 

37 3 
2) 2} 
| | | | | 
0 0 — 
0 2 4 0 2 4 


We will compute the convolution of the two discrete time signals using the 
conv function. The result of the function y = conv (x, h) is a vector, y, the length 
of which is M + N — 1, where M is x-length, and N is h-length. So, it holds 
length(y) = length(x) + length(h) — 1. We choose to design the output at a time 
interval, which is twice as that of the input and the impulse response. 


y=conv(x,h) ; 

stem(-2:8,y) 

axis([-2.5 8.5 -.5 16.5]); 
legend('y[n] ') 
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15,7 


107 


— yin) J 


hh. 


al 


Second suggested solution: 


2 


4 


6 


8 


The second method is equivalent to the first one and is based on the 2nd 
convolution principle: If the non-zero values of x[n] are contained within 
the time interval [M,, N,], and the non-zero values of h[n] are contained 
within the time interval [M,, N,], then the non-zero values of y[n] will be 


contained within the time interval: [ 


wu +My, N,+N,]. The convolution of 


only the non-zero parts of the two signals is more convenient to be calcu- 
lated. The output is then transferred to the appropriate time interval and is 


plotted. 


The following code computes and designs the convolution of two discrete-time 


signals. 


n=-10:10; 
m=length (n) ; 
k=find(n==0) ; 
x=zeros (1,m) 
h=zeros(1,m)j; 
y=zeros (1,m) 
x(k:k+4)=[1,2,3,4,5]; 
h(k-1:k+1)=[1 2 1]; 


1 


t 1 


yt=conv (x(k:k+4) ,h(k-1:k+1)); 


y (k-1:k+5)=yt; 
stem(n,y) ; 


axis([-2.5 8.5 -.5 16.5]); 


legend('y[n] ') 
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15+ | | = yin] 

10; | 

OL? I | o_o? 
-2 0 2 4 6 8 


The output signals are equivalent. 
2.14.28 Compute the convolution of the signals h[k] = 0.5", 0 <k < 10 and x{k] = u[k] —u[k — 4]. 


SOLUTION: After defining both of the signals in the same time-domain, we can com- 
pute their convolution using the conv function. 


k=0:10; 

u=ones (size(k)) ; 
k1=0:3; 
u4_l=zeros(size(k1)) ; 
k2=4:10; 


u4_2=ones (size(k2)); 
u4=[u4_1 u4_2]; 
x=u-u4; 
h=0.5.*k; 
y=conv(x,h) ; 
stem(0:20,y); 
xlim([-1 21]); 
legend('y[n] ') 


2 


— yin] 


a i 


10 15 20 
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2.14.29 Consider a system described by the following difference equation: y[n] = 


a. 
b. 


c. 


—yl[n — 1] —0.5y[n — 2] + 0.2x[n] + 0.1x[n — 1] + 0.1x[n — 2]. Compute and design for 
0<n<50: 


the impulse response of the system 
the step response of the system 
the response of the system when the input signal is: x[n] =n -0.9",0<n <5. 


SOLUTION: 


a. 


z 


The impulse response of the system will be calculated in MATLAB with the com- 
mand filter. 


a=[1 10.5]; 
b=[.2 .1 .1]; 
d=[1 zeros(1,50)]; 
h=filter(b,a,qd); 
stem(0:50,h); 


0.3 


0.2 


The step response of the system is computed and designed using the following 
code: 


a=[11 0.5]; 

b= [Pe2) 2. eal? 
u=ones (1,51); 
usr=filter(b,a,u); 
stem(0:50,usr) ; 


0.2 
0.15 
0.1 
0.05 
0 5 10 15 20 25 30 3 


5 40 45 50 
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c. The response of the system when the input signal is x[n] = 1 - 0.9",0 <n <5, is 
computed and plotted as follows: 


a=[1 10.5]; 

b=[.2 <4. 22) 
nx1=0:5; 
xl=nx1.*(0.9.*nx1) 
nx2=6:50; 
x2=zeros (size (nx2) ) 
x= [x1 x2'] > 
y=filter(b,a,x); 
stem(0:50,y); 
legend('y[n]'); 


0.5 1 


0.4 


0.3 


0.2 


0.1 


0 10 20 30 40 50 


2.14.30 Based on the following figure, and given that the impulse responses of the sys- 
tems S,, S, and S, are h,[n] = [2, 3, 4], hp [n] = [-1, 3, 1] and h,[n] = [1, 1, -1] for 0 < 
n < 2, respectively, find: 


a. The total impulse response, /i[n], of the system 
b. The output of the system when the input signal is x[n] = u[n] —u[n — 2]. 


System S1 
I t Output 
si C+ System S3 


SOLUTION: 


a. For S, and S, (connected in parallel), using the distributive property, it is: h,. = h, + 
h, = [2, 3, 4] + [-1, 3, 1] = [1, 6, 5]. 
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For the connection in series and using the associative property, we have: 
h=hyy * hy. 

Thus, using the following code, we can compute the total impulse response, 
h(n). 


hi2=[1 6 5]; 
h3=[1 1 -1]; 
h=conv (h12,h3) 


The total impulse response h(n): h= 1 7 10 -7 -5 
b. The output signal results from the convolution of the input and the total impulse 
response. 


x=[1 100 0]; 
y=conv (x,h) ; 
stem(0:8,y); 
legend('y[n]'); 
xlim([-1 9]); 


20 


— yin] 


15) 


10; 


2.14.31 For the system with the difference equation, y[n] —1.ly[n — 1] + 0.5y[n — 2] + 
0.3y[n — 4] = 0.5x[n] —0.2x[n — 1], find: 


a. The impulse response, h[n], of the system for the time interval 0 <n < 10. 
b. The output of the system when the input signal is x = [5, 1, 1, ,1, 0, 0, 1, ,1, 1, 0, 0] 
using the MATLAB functions conv and filter. 


SOLUTION: 
a. Using the following code, the impulse response of the system is computed and 
designed: 
n=O7 10% 
a=[1 -1.1 0.5 0 0.3]; 


b=[0.5 -0.2]; 
x= (n==0) ; 
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h=filter(b,a,x); 
stem(n,h) 
xlim([-.5 10.5]); 
legend('h(n) ') 


0.6 


— hin] 


0.4) 


“Lh | 


tT 


(o) 


b. Since the impulse response, h[n], is now known, it can be used to compute the 
output of the filter. We shall use the MATLAB function conv: 


x= [5 222.0 0-1 2 1 0-0) 
yl=conv (x,h) ; 
stem(0:20,yl1); 

xlim([-.5 20.5]); 
legend('y[n]'); 


— yin] 


0 5 10 15 20 


In order to get 21 output terms from the filter function (since we have 21 out- 
put terms from the conv function), we have to redefine the input signal x[n] for 


Discrete-Time Signals and Systems 109 


0<n< 20,i-e.,, to fill it in with zeros. This is done with the command x(21) = 0. 
This command adds zeros up to the 21st position of the x vector. Adding zeros 
to x gives the filter command the extra information it needs in order to con- 
tinue its calculations further. 


a=[1 -1.1 0.5 0 0.3]; 
b=[0.5 -0.2]; 

x=[5 111001110 0]; 
x(21)=0; 
y2=filter(b,a,x); 
stem(0:20,y2) ; 

xlim([-.5 20.5]); 
legend('y[n]'); 


We notice that there is a match between the two results but only with respect 
to the first 11 terms. The result produced from the use of the filter (y2) is per- 
fectly correct but y1 is partly incorrect. Our system has an impulse response of 
infinite length. However, we approximated it (we could not do anything dif- 
ferent!) by using only 11 terms when calculating the impulse response of the 


if I 


Using conv, 11 output terms will be computed with “absolute accuracy,” 
but the rest will be incorrect. In other words, the conv function ‘considered’ the 
system to be of finite impulse response with length 11. It is concluded that, if 
we try to compute the output based on the impulse response, in the case of the 
systems with infinite impulse response (IIR systems), we will surely get incor- 
rect results because of the approximation we have to do when determining the 
number of its terms. 


In the case of finite impulse response systems (FIR systems), described by 
difference equations, the use of the conv and filter functions is equivalent when 
computing the output. 
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2.14.32 Let the input sequence: 


1, 10<n<20 
x[n] = : 
0, elsewhere 


Design the output of the system when its impulse response is: 


n, -O<ns<5 
h[n]= : 
0, elsewhere 


SOLUTION: 
First suggested solution: The two signals must by defined at the same time 
: 0, -5<n<9 ,~s<snsd 
domain, thus, x[n]= and h[n]= i ‘ 
1, 10<n<20 0, 6<n<20 
The code in MATLAB is: 
nil=-5:9; 
xl=zeros(size(nl)); 
n2=10:20; 
x2=ones(size(n2)); 
x=[xl x2]; 
nl==5:5s 
hi=nl1; 
n2=6:20; 
h2=zeros(size(n2)); 
h=[hl1 h2]; 


y=conv (x,h) ; 

stem(-10:40,y) 
ylim([-16 16]) 
grid 


Since the input signal, x[n], and impulse response, h[n], are defined over 
the whole time interval —5 < n < 20, we must fill them in with zeros (where 
needed). 
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Second suggested solution: 


If the non-zero values of x[n] are contained within the time interval [M,, 
N,], and the non-zero values of h[n] are contained within the time interval [M,, 
N,], then the non-zero values of y[n] will be contained within the time interval 
[M,, + M), N, + Nj]. 


Therefore, the non-zero values of the output y[n]= x[n]* h[n] will be limited 
to the interval [5,25]. 


The code in MATLAB is: 


nx=10:20; 

x=ones (size (nx) ) ; 
nh=-5:5; 

h=nh; 

y=conv(x,h) ; 
stem(5:25,y) 

axis([-10 40 -16 16]); 
grid; 

legend('y[n]'); 


157 


—e yinj} | 


We notice that the two output figures are identical, so, the two suggested 
solutions are equivalent. 


Taylor & Francis 
Taylor & Francis Group 


http://taylorandfrancis.com 
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z-Transform 


3.1 Introduction 


Analog systems are designed and analyzed with the use of Laplace transforms. On the 
other hand, discrete-time systems are analyzed by using a similar technique, called the 
z-Transform. 

The basic lines of reasoning are the same for both cases: After determining the impulse 
response of the system, the response of any other input signal can be extracted by simple 
mathematical operations. The behavior and the stability of the system can be predicted 
from the zeros and poles of the transfer function. 

As Laplace transform converts the differential equations into algebraic terms, with 
respect to s, the z-transform converts the difference equations into algebraic terms, with 
respect to z. Both transforms are mapping a complex quantity to the complex plane. It is 
noteworthy that the z-plane (i.e., the domain of z-transform) is structured in a Polar form, 
while the s-plane (ie., the domain of Laplace transform) is structured in a Cartesian form. 


3.2 From Laplace Transform to z-Transform 


z-Transform facilitates significantly the study and design of nonlinear time-varying 
discrete-time systems, because it transforms the difference equation that describes the 
system in an algebraic equation. 

In Figure 3.1, the procedure followed by using the z-transform is illustrated, where there 
are three steps to solve the difference equation (D.E.). The direct solution of the given D.E. 
through higher mathematics is much more laborious. 

To show that z- and Laplace transforms are two parallel techniques, the Laplace trans- 
form will be used, and making use of it, the mathematical expression of z-transform will 
be developed. 


-+co 
Laplace transform is defined as X(s)= } x(t)-e* dt, where s is a complex number. 


t=—co 
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Difference Z.TRANSFORM Algebraic 
Equation Equation 
(n-plane) (z-plane) 


|.Z.TRANSFORM Solution of 
Algebraic Eq. 
(z-plane) 


Solution of 


Difference Eq. 
(n-plane) 


FIGURE 3.1 
Solution of D.E. via z-transform. 


By substituting s = o + jw one can reach to an alternative form of X(s) function, which is 
+o 


X(0,@)= i x(t)-e -e dt. Substituting in the latter expression e“' = cos(f) — j sin(ot) 
t=—co 


yields: 


X(6,0)= } x(t)-e-* -[ cos(ct)— jsin(ct) dt => 


t=—0o 


X(¢,0)= } x(t)-e-* cos(wt)dt — j | x(t)-e-% sin(wt) dt 


t=—c0 f=—c0 


(3.1) 


According to Equation 3.1, x(f) is analyzed in sine and cosine waves, whose ampli- 
tude varies exponentially in the time domain (because of the existence of the term e~). 
Each point of the complex s-plane is determined by its real and imaginary part, i.e., the 
parameters o and w. At each point of the s-plane, the complex quantity X(o,w) can be 
calculated. 

The real part of X(o,w) arises by multiplying the signal x(f) with a cosine waveform of fre- 
quency w whose amplitude decreases exponentially with a rate, o, and then is integrated 
for all instances. Thus: 


Re{X(c,0)} ~ | x(t)-e-* -cos(wt) dt (3.2) 


t=+00 


z-Transform 115 


Likewise, the imaginary part is obtained by multiplying the x() signal with a sine wave- 
form of frequency w, whose amplitude decreases exponentially with a rate, o. Hence, 


3m{X(o,0)} ~ i x(t): - sin(wt) dt (3.3) 


t=-oo 


Based on the above representation of the Laplace transform, one can produce z-transform, 
ie., the corresponding transform relation for discrete signals in three steps: 


Step 1: The first step is the most obvious: By changing the signal from continuous 
to discrete, i.e., x(f) > x[n], and, of course, by replacing the integral with a sum. 


Therefore: a > » 50, 


foo 


X(o,@)= > x[n]-e7™" ei" (3.4) 


n=—co 


Despite the fact that x[n] signal is a discrete one, X(o,w) is continuous since o and 
w variables can take continuous values. 

In the case of the Laplace transform, one could move up to any point (¢,w) of the 
complex plane (not quite any point; the integral will not converge for points not 
belonging in the region of convergence) and define the real and imaginary part of 
X(o,@) by integrating over time, as previously explained. 

If the case of the z-transform, one may, again, move up to any point of the com- 
plex plane, but instead of integration, one may work with summation. 

Step 2: In the second step, polar coordinates are introduced to represent the exponen- 
tial e-°". 

The exponential signal y[n] = e~°" can be written as: y[n] = 1", where, apparently, 
the substitution e’ = r has been implemented, which yields o = Inr. 

It is noteworthy that when: 


e y(n] =e", y[n] increases with time when o < 0. 


¢ y(n] =r", y[n] increases with time when r < 1. 
e y[n] =e", y[n] decreases with time when o >0. 
¢ y(n] =1", y[n] decreases with time when r > 1. 
¢ y(n] =e, y[n] remains unchanged when o = 0. 
¢ y(n] =r, y[n] remains unchanged when r = 1. 


Hence: X(r, @) = Sy ala ee (3.5) 
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Step 3: The substitution z = r - e/® is implemented, therefore the standard form of 
z-transform arises. 


X(z)= Yan) Zz (3.6) 


Equation 3.6 forms the definition of bilateral z-transform. 

Because there are both positive and negative z-powers, convergence is, in some cases, 
possible inside the region of a ring of the complex plane. A serious difficulty with this 
transform is that, in order to be able to return to time domain, the convergence region must 
also be known. For casual systems, the unilateral z- transform is defined by Equation 3.7. 

It is a power series of negative z-powers and convergence is possible if |z| is large enough. 


nN=co 


X(z)= Yat] zm (3.7) 


n=0 


For the convergence region it is true that: 


1. A sequence of finite-length has a z-transform for which the convergence region is 
the entire z-plane, except perhaps from the points z = 0 and z = oo. z = oo is included 
when x[n] = 0 for 1 < 0 is true and point z = 0 is included when x[n] = 0 forn > 0 is 
true. 

2. A right-sided sequence has a z-transform for which the region of convergence 
(ROC) is the exterior plane of a ROC circle: |z|a. 

3. A left-sided sequence has a z-transform for which the region of convergence (ROC) 
is the interior plane of a ROC circle: |z| < b. 


z-Transform (Eq. 3.6) is a valuable tool for analyzing discrete Linear Time-Invariant (LTT) 
systems. It can be used for: 


e Efficient calculation of the response of an LTI system (convolution in the n-plane 
y(n) = x(n)*h(n) is computed as a product in the z-plane: Y(z) = X(z)H(2), so y(n) = 
IZT(Y(2). 

¢ Stability analysis of an LTI system (through calculation of the ROC). 


¢ Description of LTI system with regards to its behavior in the frequency domain 
(low-pass filter, band-pass filter, etc.). 


3.2.1 Comparison of the s- and z-Planes into the Region of Convergence 


The main differences between the s- and z-planes are presented in Figure 3.2. 

The points of s-plane are described by two parameters: o parameter, which corresponds 
to the real axis and determines the rate of reduction of the exponential, and w parameter, 
which corresponds to the imaginary axis, which determines the oscillation frequency. 
Both parameters have a rectangular arrangement on the s-plane. 
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jw F-—~~} 


Re 


FIGURE 3.2 
s-plane is Cartesian whereas z-plane is polar. 


The resulting geometry arises from the fact that for each s number, the position is deter- 
mined by the relation: s = o + jo. 

The z-plane is polar. Each complex number z is determined by its distance, r, from the 
origin, which corresponds to the reduction rate of the exponential (recall the expression 
X(r, @)), and by w parameter, which corresponds to the angle between r and the positive 
horizontal semi-axis. The z-plane is structured in polar coordinates, and this is interpreted 
by the equations: z = r-e/” or z = re/” = r(cos w + j sin @). 

As a consequence of the above differences, the vertical lines in the s-plane become circles 
in the z-plane. This is due to the substitution o = In r, which has previously been imple- 
mented. For instance, the imaginary axis of the s-plane, i.e., the line formed by setting o = 
0, will correspond to a circle with a radius r = 1 into the z-plane. Indeed, the relation o = 0 
corresponds to o = Inr in the z-plane, which yields r = 1. 

Lines parallel to the imaginary axis located in the left half-plane (6 < 0) correspond to 
concentric circles, which are located within the unit circle in the z-plane and, likewise, for 
lines located in the right s-half-plane. 


3.3 Properties of z-Transform 


The most important properties of the z-transform, commonly used for problem solving, 
are presented in the following sections. 


3.3.1 Time Shift 


If X(z) denotes the z-transform of the casual sequence x[n], then the corresponding trans- 
form of x[n — N] is given by z-VX(z). Time shift adds or subtracts the axes’ origin or infinity 
from the region of convergence of X(z). This is one of the most important properties of the 
z-transform since it allows difference equations with zero initial conditions to be expressed 
as functions of z and to calculate the output of a system regardless of the type of input. 


This Then... 


x[n] < X(z) x[n — N] © z-NX(z) 
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In the case of non-zero initial conditions: 


If... Then... 


x[n] > XZ) x[n—-N] oz‘ X(z)+4 Sx jz 


i=l 


Proof: Let y[n] = x[n — N], then Y(z) = y X[n-N]z" 


n=—00 


Assume k = n— N thenn =k + N, and substituting in the above equation, we have: 


co 


Y(z)= pa x{kjz% =2-°X[z] 


k=-c0 


3.3.2 Linearity 


Let x[n] be a function, which is produced from the linear combination of two functions, 
x,[n] and x,[n], with regions of convergence P, and P,, respectively. The region of conver- 
gence of x[n] includes the intersection of P, () P3. 


This Then... 


x[n] — X(z) ax,[n]+ bx,[n] pan aX, (z)+ bX, (z) 


3.3.3 Time Reversal 


If the z-transform of the x[n] function is X(z), with a region of convergence, P, then the 


transform of x[-n] is X (2) with a region of convergence 7 
Z 


Proof: If we have x(n) & X(z), then x*[-n] <~—> X* (1/2*) 
Let y[n] = x* [-n], then 


v= Srtnie"=| Sete”) -{ Seat)" = x*(1/2*) 


n=—00 
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If the ROC of F(z) is rg < |z| <1, then the ROC of Y(z) is 


% . 1 1 
Tp <|L/z*\ <1, ie., —>\|zl>— 
TR t 


Moreover, it is easy to show that: 


x[-n] 2» X(1/z) and ROC is Jee: 


TR qt, 


3.3.4 Convolution 


Let two functions x,[n] and x,[n] with corresponding z-transforms and regions of conver- 
gence x,[n] — X,(z) when z € P, and x,[n] + X,(z) when z € P,. The transform of the convo- 
lution of the signals x,[n] and x,[n] is x,[n] * x,[n] — X,(2) X,(z) = X(@), where the region of 
convergence of X(z) is equal or contains the intersection of the regions of convergence of 
X,(z) and X,(z). 


Tf..3 Then... 
x[n] > X(z) x,[n] * x,[n] — X,(z) X2(z) 


0 


Proof: The convolution of x,[n] and x,[n] is defined as x[n]= x,[n]* x,[n]= > x,[k]x,[n—k] 


k=-c0 


n=—00 n=—e0 |_ n=—co 


The z-transform of x[n] is X(z) = y, x[n]z" = y | y, x,[k]x,[n- af 


By interchanging the order of the summation and applying the time-shift property, we 


obtain X(z)= y, =p x,[n—k]z" 


k=—00 


= xy) co bs ai = X,(z)X,(z) 


k=-00 


n=—00 


3.3.5 Differentiation in z-Plane 


Let X(z) be the transform of x[n] function with a region of convergence, IH. Then, it is 


nx[n]  —z oe with the same region of convergence. 
If Then 
x[n] < X(z) nx[n] o -z ane) 
dz 
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Proof: 


0 


x(z)= by x[n]z" 


n=—00 


7, aX(2) _ >y nx[n]z-" = ¥ -nxfn” 


dz n=—0o n=—00 
-Z a) > nx[n] 
dz 


3.3.6 Multiplication by an Exponential Sequence 


Let X(z) be the transform of x[n] function with a region of convergence, P. Then, 


zyx[n]o X (2} The consequence is pole and zero locations are scaled by Zp). If the 
Zo 

ROC of FX(z) is rR < |z| < rL, then the ROC of Y(z) is rR < |z/z0| <rL,ie., |z0|rR < |z| < 

|zO| rL. 


Proof: Y(z)= Dy zyx[n]z "= Yxtof 2) 7 x(2 
0 0 


n=—°o n=—90 


3.3.7 Conjugation of a Complex Sequence 


If we have x(n) = X(z), then x*[n]<—— X*(z*), and ROC = R; 


Tfecs Then... 


x[1] & X(z) x*[n]} > X*(z*) 


Proof: Let y[n] = x*[n], then 


°° 


Y(z)= »y x*[n]z" = » ante" = X*(z*) 


n=—00 
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3.3.8 Initial and Final Value Theorem 
If x[n] = 0 when n < 0, then x[0]= lim X(z) (Initial Value Theorem) 


For causal stable systems, it is true that: 


x[c°] = lim x(n) = lim(2— 1)X(z) (Final Value Theorem) 


If... Then... 
x[n] — X(z) x[0] = lim X(z) 
x[n] < X(z) x[co] = lim x[7] = lim(z —1)X(z) 


3.3.9 Correlation of Two Sequences 
If x,[n] — X,(z) and x,[n] — X,(z) 


co 


Paya tM) = YY 34 (n)xe_(n— m) > X, (2)X,(27) 


N=—0e 


The following is the correlation of two sequences. 


PxX_(m) = x(a, [(n-m)] 


n=—0o 


Or, rearranging the terms, we get 1%x,(m)= y X4(11)X5 [-(m- n) |. 


Thus, the right-hand side of the second equation represents the convolution x,(1) and 
x,(-m) and can be written as 


1XX(m) = x,(m) ® x,(—m) 


Z[1x,X2(m)] = Z[x,(m) @ x,(—m)] 
Z[rx,x,(m)] = Z[x,(m)] ® Z[x,(—m)] 
Z[x,(m)] = X,(z)and z[x,(m)] = X,(z") 
Z[rx,x,(m)] = X,(z)X,(z") 


Dr! 


3.4 Inverse z-Transform 


Applying the z-transform results in the transfer from the discrete-time domain to the 
z-domain. The inverse procedure is carried out using the inverse z-transform. 
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The inverse z-transform is defined by 
x[n]=Z7 [ x(z) ] = aj PO” dz (3.8) 


where c is a circle with its center at the origin of the z-plane, such that all poles of X(z)z"" 
are included in it. This integral can be derived by Cauchy’s residue theorem. 

Since the calculation of the integral is quite difficult, the use of matrices is suggested. In 
this way, the time functions of fundamental complex functions are derived. The majority 
of these matrices, however, can cover only a few cases. Therefore, other methods are used 
to calculate the inverse z-transform. 

There are three methods for calculating the inverse transform of a function, X(z): 


a. Power series expansion 
b. Expansion partial fraction 
c. Complex integration (through the residue theorem) 


3.4.1 Method of Power Series Expansion (Division Method) 


Using this method, specific samples of the inverse z-transform are calculated, but a corre- 
sponding analytical expression is not provided. By dividing its numerator by the denomi- 
nator, function X(z) forms a series with respect to z. 


3.4.2 Method of Partial Fraction Expansion 


Partial fraction expansion is a particularly useful method for the analysis and design of 
systems because the effect of any characteristic root or eigenvalue becomes obvious. 


It is helpful to use a 
Z 


and not X(z) while applying the method of partial fraction 


expansion. We distinguish three cases of partial fraction expansion for 


X(z) according to 
the form of its poles. n 


¢ Case of distinct real poles: 


In this case, X(2) is expanded in a fractional sum series, as follows: 
Zz 
X(z) B(z) Cy é 


= = a4 u 3.9 
2 @—p).2—Pp,) &—P)” * @—p,) ea 


The c; coefficients are computed using the Heaviside formula for distinct poles, 
hence: 


- BG) 
c= tin Pepe ey teed (3.10) 
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¢ Case of non-distinct real poles (multiple real poles): 


In this case, X@) is expanded in a series of fractions, as follows: 
Zz 
X(z) _ B(z) _ i, Cio 
Z z—p,)"..(z—p,) (2-p) (z-p,y 
(Z—p,)"...(2— pn) Pi) (Z-p) @.t1) 
ae Cin Co Cy 


+ + 
(z—p,)"  (Z-pr) (2-P,) 


The c, coefficients are computed using the Heaviside formula for multiple poles, 
hence: 


Cc 


; 1 im qi") Gay x0} (3.12) 


~ (n= jylenn Lazo 


The remaining coefficients are computed using Equation 3.10. 
¢ Case of complex roots: 


In this case, the coefficient of the numerator for one of the complex roots is com- 
puted using Equation 3.10 or 3.12. Therefore, the coefficient that is the numerator 
of the term, which has the conjugate root of the former at the denominator would 
be its corresponding conjugate. 


3.4.3 Method of Complex Integration 


This method is quite general and is being used when one or more partial fractions of the 
expanded F(z) are not included into the lookup tables of z-transform pairs. 

This method relies on the use of the definition formula of the inverse z-transform. 

In order to use Equation 3.7, the application of the residue theorem is required, which is 
given by: 


PFtee"" dz= 2nj>, residues [F(z)z""" | (3.13) 


In the latter expression, also known as Cauchy’s formula, )} represents the sum of resi- 
dues of all the poles of F(z), which are included in the c curve. 
Combining the above expressions, we obtain: 


finl= SY! residues| F(z)z""*] (3.14) 


e Ifthere is a simple first-order pole of F(z) - z"" (i.e., z = a), then its residual is given by: 


F(z)z" \(z-a) (3.15) 


z= 
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e If there is an m-order pole of F(z) - z"', then its residual is given by: 


1 q-) 


Ga Diagn Pee — a)” (3.16) 


Z=a 


3.5 z-Transform in System Analysis 


z-Transform gives a significant amount of possibilities in system analysis for: 


e Effective computation of the response of a discrete LTI system (the convolution in 
the discrete-time domain y[n] = x[n]*h[n] is calculated as a product in the z-transform 
domain). 


e Analysis of the stability of an LTI system (through the calculation of the conver- 
gence region). 
e Characterization of an LTI in relation to its behavior in the frequency domain. 


3.5.1 Transfer Function of Discrete-Time Signal 


The transfer function is defined as the quotient of the z-transform of the output of an LTI 
system to the z-transformation of its input, when the initial conditions are zero, and cor- 
responds to a relation which describes the dynamics of the particular system. 


melts) 


EA a) 


eH (3.17) 


The transfer function is the z-transform of the impulse response. Equation 3.17 shows 
that the response of the discrete system can be calculated by the relation: 


yln] = IZT(X(z)H(2)) (3.18) 


Let the input of an LTT system be a signal described by the impulse function. The z-transform 
of the impulse response hn] is the transfer function of the system. The transfer function 
does not depend on the waveform of the input but only on the system characteristics. 

In the case that the variable z is in the unit circle, i.e, when z = 2, then the transfer func- 
tion is the same as the system response frequency, provided that the unit circle belongs to 
the region of convergence. 

Moreover, if, at the input of a discrete-time LTI system, the complex exponential sequence 
x[n] = z" is applied, then the output will be H(z)-z". That is, the sequences of z” form, are 
eigenfunctions of the system with eigenvalue H(z). 


3.5.2 Causality of Discrete-Time Systems 


We can examine whether an LIT system is casual by studying its impulse response. 
According to the definition, a discrete-time LTI system is casual when the impulse response 
is zero, i.e., h[n] = 0 for each n < 0. 
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According to the condition of causality, the impulse response of a causal system is zero 
for each negative value of the variable n; therefore, the impulse response is a clockwise 
sequence. Then, the region of convergence of the transfer function will be in the form of 
an area outside of a circle. Based on the definition a causal sequence, /i[n], corresponds to 


a function of the form H(z)= Y }Jn]z~™, i-e., H(z) does not include positive exponents of z, 
(z) [n] P Pp 


and consequently, the range of convergence includes infinity. 

A discrete-time LTI system is casual if and only if the region of convergence of the trans- 
fer function is part of the complex plane exterior to the circle passing through the most 
distant (from the origin) pole and includes infinity. 

In the case of discrete-time signals, the shift by N in the time domain corresponds to 
the multiplication times z-’ in the z-domain. That is, if x[n] + X(), then x[n —N] < z-NX(@). 
If function X(z) is rational, then the z-transform of the displaced sequence will also be a 
rational function of z. From the theorem of the initial value: h[0]= lim H(z). If H(z) is not 


rational with respect to z, the latter relation is true only if the rank of the numerator is 
smaller than or equal to the rank of the denominator. 
Thus, an LTI system with a rational transfer function is casual when: 


e The region of convergence is outside of the circle defined by the origin and the 
most distant from the origin pole. 


e The rank of the numerator is smaller than or equal to the rank of the denominator. 


3.5.3 Stability of Discrete-Time Systems 


An LTI discrete-time system is stable when the impulse response h[n] converges, i.e., when 


< oo (3.19) 


n=—eo 


A conclusion of great practical value is derived from the comparison of the condition 


> h[n] < with the transform of the sequence /[] From the definition of the z-transform, 


too 00 n=—00 


y h[n]|/< ce of stability is equivalent to the requirement for H(z) to be finite when|z|= 1. 


Or, in other words, the region of convergence should include the unit circle. 

Therefore, an LTI discrete-time system with an impulse response hn] is stable if and 
only if the region of convergence includes the unit circle. 

A causal LTI system with a transfer function H(z) is stable when the poles are included 
in the unit circle, i.e., their magnitude is less than one. The closer to the center of the circle 
the poles are, the greater the relative stability of the system is going to be. 

The stability of the output of an LTI system depends both on the stability of the input 
signal and on the stability of the system. Under certain conditions, an unstable system may 
produce a stable output signal. 
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A system is practicallyfeasible when it is both stable and casual. Such a system will 
have a transfer function with a region of convergence of the form |z|> a, where 0 < a < 
1. Consequently, all of the poles of H(z) must be included in the unit circle. 


3.5.4 Transfer Function of Connected Systems 


The transfer function of a discrete system can be considered to be produced by the connec- 
tion of other simpler systems and vice versa. 


1. Systems connected in series (illustrated in Figure 3.3) 
In this case the total transfer function of the equivalent system is equal to: 


H(z) = H,(z)-H,(z) (3.20) 


2. Systems connected in parallel (illustrated in Figure 3.4) 
In this case, the total transfer function of the equivalent systems is equal to: 


H(z) = H,(z)+ H,(z) (3.21) 


3. Feedback connection of systems (illustrated in Figure 3.5) 
In this case, the total transfer function of the equivalent system is: 


Hi, (2) 


ty 


(3.22) 


FIGURE 3.3 
Systems connected in series. 


FIGURE 3.4 
Systems connected in parallel. 
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U(z) 


H,(Z) 


eae) eee ee ee ee 


FIGURE 3.5 

Feedback connection of systems. 

3.5.5 Transfer Function of Discrete-Time Systems 

In the digital signal analysis, a meen system moet is described by a difference equation, 
the general form of which is: Yas [n—-k] aye [n—k]. Applying z-transform to both 


k=0 k=0 
parts of the difference equation produces the system transfer function. 


YOY a, 2° = X(2) > bz (3.23) 
k=0 = 
Thus, 
Yaz 
H(z) = ~@ = 20 __ (3.24) 
a Yaz 


From the transfer function H(z), the following can be derived: 


1. The impulse response, h[n], by applying the inverse z-transform. 


2. The frequency response, H(e!”), by substituting z = e/”, where @ is the digital fre- 
quency in the interval (0, z). 


_ : 
Y(ei# he 
H(eiv) = TED) «40 (3.25) 
X(e! ) Da e7 jok 
k 


In a discrete system, the transfer function is defined by: 


He”) = y h{njei" (3.26) 
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H(e/”) is periodic with a period 22 and determines the effect of the system on the fre- 
quency content of the input signal. 

If an input signal x[n] = e”" is applied to a discrete system with an impulse response, 
h[n], then the output will be: 


co co 


yl] = xr] * hn] = hn] *x{n] = "S° hk k] =>” hlkye"™? 


k=-00 k=—c0 


y(n) =e" H(e") =e" He") ZH(e"”) (3.27) 
That is, the output, y[n], will be equal to the input, x[n] = e/°", multiplied by a weight func- 
tion, H(e’”). 
As already mentioned, the frequency response of an LTI discrete-time system results 
from the transfer function H(z) by substituting z = e/”. This substitution is allowed when 
the following are true: 


too too 
e The sum Ea) must be finite, i-e., Y |xtn] <0 


e The region of convergence must include the unit circle. 


The study of the frequency response involves the calculation and plotting of its ampli- 
tude and phase. Quite often the group delay is studied, which is defined as: 


T(@) =- oy (3.28) 


An LTT discrete system is called of linear phase when the frequency response is of the 
following form: 


H(e") =|H(e!”)\e (3.29) 


where a is a real number. Linear phase systems have a constant group delay t(w) = a. 
Consequently, the group delay (which indicates the delay that the input signal suffers from 
the system) is the same for all w values. 
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3.6 Formula Tables 


The z-transforms of the fundamental functions and the properties of z-transform are given 
in the following Tables 3.1 and 3.2, respectively. 


TABLE 3.1 


z-Transform of Fundamental Functions 


x[n] X(z) ROC 
1 6[n] 1 Allz 
2 u[n] 7 |z| >1 
z-1 
Zz 
3 —u[-n-1] |z| <1 
z-1 
4 d[n — m] zm All z-plane except from 0 
(m > 0) or - (m <0) 
2 a"u[n] a z|>|a 
z-a 
Zz 
6 —a"u[-n — 1] z| <|a 
z-a 
7 na"u[n] ae z|>|a 
(z-ay? 
8 [-n-1] e < 
—na"u[-n — Zz a 
(za)? 
2 
9 (n + 1)a"u[n] z z|>|a 
z-a 
2 
Q 
10 (cos Qn)u[n] ee z|>1 
z —(2cosQ)z+1 
11 (sin Qn)u[n] 22 z|>1 
2? —(2cosQ)z+1 
2 = 
12 (r’cos Qn)u[n] = SaaS z|>r 
2? —(2rcosQ)z+r? 
13 (r’sin Qn)u[n] __renQ)z z|>r 
2? —(2rcosQ)z+1r? 
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TABLE 3.2 


Properties of z-Transform 
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Property Function Transform ROC 
x[n] X(z) R 
x,[n] X,(z) R, 
x2[n] X2(2) R, 
1 Linearity ax,[n] + bx,[n] aX,(z) + bX,(z) R’'DR, NR, 
2 Time shift x[n - N] ZNX(z) R’DR 
3 Multiplication with z/ zix{n] ( : R’= |Z|R 
xX| — 
Zo 
4 Multiplication with e@" e2rx[n] X(e7°Z) R’=R 
fe) Time reverse x[-n] 
x({t Ris 
Zz R 
6 Multiplication with n nx[n] dX(2z) R’=R 
~ dz 
7 Sum n R’'DRn{|z| >3} 
X(z 
> x[n] 3 (z) 
aes 
8 Convolution x,[n]*x,[1] X,(z)-X,(z) R DR NR, 
Initial value If x[n] = 0 when n <0, then x[0] = lim X(z) 
10 Final value For stable systems lim x[n] = lim(z- 1)X(z) 
neo z1 
——————— 
3.7 Solved Problems 
. : > 
3.7.1 Compute the z-transform of unit-step function: u[n] = 1 when 120 
0 when 1n<0O 


SOLUTION: Based on the definition of the z-transform, the following is calculated: 


+ 


00 


: u[n].z-" = 


x=) 


_ = ; 
2 Lie =e 


The unit-step function affects the limits of summation. 


+oo 


(1) 


Say is a geometric series (with first term 1 and ratio z~'). Consequently, it 


n=0 


must be true that |z|<1 or |z|>1. 


The condition |z|>1 defines the region of convergence of the transform, ice., 
the set of values of the complex z-plane, for which the sum of the z-transform 


converges. 
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3.7.2 Compute the z-transform of the function: x[n] = a"-u[n]. 
SOLUTION: Based on the definition of the z-transform, the following is calculated: 


X(z)= y a"u[n]-z" = ya -zty 
n=0 


n=—0o 


The unit-step function restricts the summation limits from n = 0 to infinity. 
The term (a - z"!)n is the general expression of a geometric progression with: a, = 1 


A B 
and N(z)= =e 
Z--— Z- 
2 3 
In order for Yay to converge, the geometric progression should be 
n=0 


decreasing, ie, |a-z4|<1. 
This constraint yields: 


= Sec 1 1 z 
a-Z = = = 1 

Dy ) 1-az! ta Z-a () 
Zz 


n=—00 


Hence: x[n] = a" -u[n] @ X(z)= (2) 


The above transform is true only when: |a|-|z-'|< 1 or |a|<|z| or |z|>|a| 
The condition |z|>|a| defines the region of convergence of the transform, ie., 
the set of values of the complex z-plane for which the z-transform converges. 


3.7.3 Compute the z-transforms of the functions: x,[n] = 6[n — 2] and x,[n] = 6[n + 2]. 


SOLUTION: Functions x,[n] and x,[n] are two time-shifted impulse functions. 


Function x,[n] = 6[n — 2] is equal to zero except when n = 2. 
n=+00 


The z-transform is X,(z)= Sala =z and has a double pole at z = 0. 


n=—co 


The region of convergence is the entire complex plane except the point (0,0), yet it 
includes infinity. — 


Correspondingly, for the function x,[n], it will be X,(z) = x,[n]z" =z? with 


n=—0o 


a pole at infinity. The region of convergence is the entire complex plane including 
(0,0), yet not including infinity. 


3.7.4 Compute the z-transform and the region of convergence for each of the following 


n n+5 
functions: ainl=( 2 u[n] and vint-( 2 u[n + 5]. 
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SOLUTION: The function x[”] takes non-zero values only when n > 0, hence, it is a 
causal function. 
The function y[n] is derived by shifting x[n] by 5 time units to the left y[n] = 
x[n + 5]; therefore, there are some negative values of n for which y[n] 4 0. Thus, y[n] 
is a non-causal function. 


The transform (5), given in the Transform Lookup Table 3.1, yields X(z) = 


with a region of convergence Iz! < 6 Ye Fe 


The transform of the sequence y[n] can easily be computed using the time-shift 
property (Property 2 — Table 3.2). Y(z) is then given by: 


Viger" = 


The poles and regions of convergence of X(z) and Y(z) functions are presented 
in the following Table 3.3: 


3.7.5 Compute the z-transform and the region of convergence for each of the following 


n n-3 
functions: x[n]= (2) u[-—n—1] and y[n]= (2) u[—n + 2]. 


SOLUTION: The function y[n] arises from the time shift of x[n] by 3 time units to the 
right, y[n] = x[n — 3], hence, as shown in the figures of Table 3.4, there are some positive 
values of the variable n for which y[n] 4 0. The function y[n] is non-causal. 


From the transform of (5), given in the Transform Lookup Table 3.1, it is true 


2) 
that X(z)= —. with a region of convergence dl< L 


L—-— 
4 


The transform of y[n] can be easily computed by using the time-shift property 
(Property 2 — Table 3.2). The function is given as 


(1) 


ze 1 
cs a ae 5 
—— 2 eer 
74 2(z 5) 


The poles and regions of convergence of X(z) and Y(z) functions are presented 
in the following Table 3.4. 


n-3 
The computation of the z-transform of the function y[n]= (5) u[—n + 2] 
without using the time-shift property is given next. 
Based on the definition of the z-transform, we get: 


rey= Dyer ==-Si(F) evans SG] = 


n=—00 n=—c0 n=— 
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‘JYSII 39 0} pauep ‘Tesned-uoNy 
co = Zz pue a= Z :sajog 


9 
oe 
Zz = @A 


9 


| oe 


a 


[g+u]n (= =[u]A 


9 ‘Ayuyur apnyour 
jou saop pue = < Z| SI (Z){ JO auas1aAUuOd 


9 jo uorgar ayy, ‘AyTuTFUT sapnouT pue 
a < Z| SI (2)x Jo aoua8IaAUod Jo UOTsal ay, 


“‘}YBII UY 0} pauyap ‘Tesned 
a =Z ‘30g 
ak 


ei 


[u]n a =[u]x 


(2)X pure (Z)X I0j aUaSIaAUOD Jo suOTSay pue safog ay], 


ee JIAVL 


“UOISaI Ya] AY} UT pouTap ‘Tesneo-u0Ny 


‘0 =z pue : =Z :sojog ‘O19Z SOpNoUT HT nq i >| 
SI (Z){ JO sua8IdAUOD Jo UOTSar JU, 
E a “| e ‘O19Z apNput jou saop pue Las Z| 
=(2)A s 
T SI (Z)X JO ddUdBIBAUOD JO UOIBAI JY, 
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“UOIZaI Ia] SY} UT pauTjap ‘Tesneo-yUy 


Paz Og 
oy 
4 —=()x 
I . 
eee! | | | | LU Tttreeseaee 
[r-u-yn Gia 


(ZX pur (Z)X Joy BUaSIaAUOD JO SUOIZayY puR safog IU], 
pre T1aVi 


z-Transform 


n n=2 n 
The sum » Gal is written as > Gal 


n=—0o n=—0o 


-1 
A more sophisticated way to build in the relation is to set Ga =W. 
ww 5 -1 ‘ k 2 1 0 -1 2 
Then, » et] Sa HWE WW + WF WW or 
yw =. +W+...W24W!4W!4W14Wor 
n=—2 
yw =W74+W14W°4+W!14W74...4WE 4+... 
n=-2 
en 4 
which is a series with the first term W~ = (E2"] and ratio W = Be 
And for decreasing geometric progression: |W|<1 or ee <lor Z| < 2, 


2 
Substituting the first term a, for W~ = Ga 


yields 
Bl “TP Ba) 5a), 
= Ys it F 
ay 4 _ \4 _ 5\4 5) 2° 
1-W 4 4 4 5 4 5 
1-—z =z-1 Z-— Za 
5 5 4 


and then W = sz into 
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Then, from relation (2), Y(z) is computed: 
n=-+00 3 n=2 n 
7 i. ‘i, B Days, 2-2. ih 
Y= ¥ ylnkz --(5] X(2 =+ = ( a (5) 
N=—00 Z-— Bl Z-— 
4 4 


n=—0o 


3.7.6 Compute the z-transform and the region of convergence for the function: 


x[n]= (2) u[n]+ 2” u[—n— 1]. 


SOLUTION: The given function can be considered the sum of x,[n] and x,[n], where 


u[n] and x,[n]=2"u[—n—1]. 
directly results from the transform (5) of Table 3.1, which 


x,[n] = 
The transform of x,[n] 
7) 1 
: Zz : : 
is x,[n]= (3) uln] is X,(z) with a region of convergence Id)> =. 
3 
x{n] (5 u[n] +2" u[—n—1] Region of convergence J eller 
3 3 
te xin] 
n 
2" u[-n-1] (1/3) un] ee lad 
, ™ 
yo 1 ‘N 
fi y 
/ 
(} f >. \ Re 
0 *— 
-10 -8-6-4 -2 0 2 4 6 810 -21 -1 07 1 % 
q y 
(n) X 44 y 
Ny id 
aa ee 
Z 


Based on transform (6), we have —2"u[—n- 1] x 
Z — 


From the property of linearity, if y[n] «= Y(z) then ay[n] — aY(z), hence 


x,[n] = 2” u[-n-1o _—*_ witha region of convergence |z|< 2. 
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Consequently, X(z) = eee ee 
22 
3 
2 —22-gotz = 
30 3 _ Zz 


= =-5 
1 1 3z* -7z+2 
[2-3 }e-2 [2-2 }e-29 


The poles of X(z) are z; = 2 and z, = > Since x[n] is bilateral, the region of conver- 


gence will be the ring between the two poles, ie., it will be defined by the relation 
; <|z| <2, which is the intersection of the regions of convergence of X,(z) and X,(z). 
3.7.7 Compute the z-transform for the function: x[n] = 1 - a"u[n]. 


SOLUTION: According to the transform (5), x[n] = a"u[n] =e X(z). The derivative 
of X(z), with respect to z, is expressed as: amid 


“( z peel ee = 


dz\z-a (z-a) ~~ gage ~~ (aaah 


Based on the property n- x[n] > —z 


me) the following is derived: 


= az 
 (z-ay 


Z{n-a"uln]} 


3.7.8 Compute the z-transform of the function: x[1] = (3 u[—n] 


SOLUTION: Based on the transform (6), we have: 


(3) u[-—n ee ee l<=. (1) 
3 


The function is time-shifted by one unit, i.e., 1 is substituted by n — 1, so: 


1 n-1 a 4 
(3) ul (n-1) aly or 


3 


138 Digital and Statistical Signal Processing 


3.7.9 Compute the z-transform of the function: x[n]= “(3 u[n—2] 


SOLUTION: According to the transform (5), we have (3 uln] oo = when |z|< > 


Based on the time-shift property, we have: a 
1 n-2 
= 2 2% 
(3) u[n-2]oz "i > 
ee —_—— 
2 
iy (ay i) 21) 1) 2" 
(5) (3) u[n— 2] < (3) -=(3) u[n—2]< ( i 
Z-— z= 
2 2 
Making use of the differentiation property: 
> 2 1 -1 
1 n-2 dl Pe or (2-3 —Z 
n| — u[n—- 2] -z 
2 dz) 4, 1 4 1/ 
a)” (ed) 
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3.7.10 Define the initial and final value of the impulse response of the system when: 


22 
Ore 


SOLUTION: From the initial value theorem: 


22" 
0] = limY(z) = lim = 0 1 
ON ee eae) " 
From the final value theorem: 
[=] = lin(z-1) Y(2) = lim __-_2 Q) 
on et z31(z—a)(z—b) (1-a\(1-b) 
3.7.11 Find the first four coefficients of the function x[n] when: 
Az 2+2z14+27 
X,(z) = ————_., X, (2) =§ for [elo 1 
1) gg +o 2(2) 1+z7 f 
SOLUTION: 
a. For X,(2): 
Ag? 2? 4z 

x(2)= (1) 


(2? -221+2)-2? ~ 272-2741 


The polynomial division of the numerator and the denominator of X,(z), yields: 


Az 227-27 4+1 
-—47+4-2z71 —2714277 4754... 
4-271 

-44 471-272 
271 + 277 
—2z71-277-275 
-2z73 


=> X,(z)=221+2z27 +274... 


Therefore, the sequence x,[1] is: x,[n] = {9 2, 2, L.} (2) 
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b. For X,(z), we provide another method for a polynomial division: 


z' +027 -3z°-42* 


z3+2244)2?-1 


ze+2+4z71 
~3-42' 
-3 =67°=127" 
—471 +627 +122° 
=4z7" = 87° — 162" 


627 +20z° +1627 


Thus, the sequence x,[n] is: 


X,(z)=z7 +02" -3z% —4z4+4... 
=> x,[n] = 06[n] + 16[n — 1] + 06[n — 2] — 36[n-3]+... i) 


3.7.12 Compute the inverse z-transform of the functions: 
a Ss 


F,(2)= “12 ay 2 
(l-az)(1-bz") = (z-a)(z-b) 


_ 2(z7 -5z+6.5) 
F,(z)= (2-2-3) 22-3) ’ for 2< Iz<3 


2z7 +3z 
Ce 
(z+ 2)(z-4) 
z°—20z 
F,(z)= Fe ANOS maak 
(z-2)°(z-4) 
SOLUTION: 
2 
a. It is true that: f,[n] = Leste 2] (1) 
The residue of the simple pole z = p is: 
2 n-1 n+1 prt 
22 oe Zz | 2 . es 
(z-a)"|,_, (z-a)|,_, (b-4) 


z-Transform 


The residue of the double pole z = a is: 


a 2? z) [n+1"(z—-b)-2""| __, [n+ 1](a-b)-a 
dz\.z—b 


a (z—by’ bee (a—b)’ 
Hence: 
flr a"| n(a—b)—b |+b"*" 
n)\ = 
(a—b)’ 
b. We expand the function F, (z) in a sum of fractions: 
2 — 
Boje el 2 448. ee BO 
(z—2)(z-3) z-2 z-3 (z-3) 
=> E(z= : : : 


+ + 
z-2 z-3 (z-3) 


= 4 3 
=i 2) if :) = :) sothat 24 <1and 74 <1 


2 Z 3 3 9 

1 2 4 8 1 5. 2 Be 1 27.32 42° 
=—|1+—+—4+-4+-- 1+—+—+—+---]+—|14+—+ + 

Zz z 2 2 3 3 9 27 9 3 9 27 


(3 2 2 23 -G z 2 2 ne 22 322 423 
=| —+>4+—+—4+-- +444 [4] 54+ 4+ 4 St 


2° 272° 3 74 3 3 39 3t Ce Ge a 
= Yrs" L (He + Yorn ly 2 
n=1 n=0 n=0 


. -1 -2 
or inversely, we have u, =2"",n21and u, =—-(n+2)3"“",n<0. 


2 
( pas ee can be written as: 
(z+2)(z-4) 
F,(z) 2z+3 A B 1 11 
= = + hen A = d B= 
z  (z+2\(z-4) z+2 z-4 oe Me - A 


Thus, 


1 z 11 z 


F;(z)= + 
N= Gop Gad 


i) 


141 


(4) 


142 Digital and Statistical Signal Processing 


and, by applying the inverse z-transform, we get: 


1 tine Lt pay 
fal = 2)" +4) 


2 —20z 


d. Fi(z)= (-2)(z-4) can be written as: 


F(z) 27-20 == A+Bz+Cz* D 
Zz (z=2) (2-4 (z=2/ z-4 


Multiplication with (z - 2)°(z — 4) yields: 


2? ~20=(A+Bz+Cz’)(z—4)+ D(z—2)°. 


We then compare the coefficients of z in the last equation, which yields: 


A=6,B=0,C= 4 andD=-y. 


F,(z) _ 2 —20 _ 640+ Yor? 4 


Thus, 
Zz (z=2)'(2=4) (z-2)° 


E,(2) = jiaz+2° i z j=} 12z+ 427 — 427 +2° i z 
a (z-2 | 2\z-4) 2 (z-2)° 5\ gaa 


1 7 ere 1 2 
2 (z-2) z-4 


meals if Z ) 


(a2) 
1 | 2(z—2)° +42? +8z i z Z 4 ee +42 i z 
“a (07 2\z-4) 2|z-2 ~ (z-28 | 22-4 


The inverse z-transform yields: 


faln] = Sle" ae 2.n’2") = 4” | —t pr-l + n22" _ g?n-l 


z-Transform 


zr4ztsi 


3.7.13 Compute the inverse z-transform of the function: F(z) = 


a. the method of the expansion of the sum of partial fractions and 
b. the method of complex integration. 


SOLUTION: a. Computation of f[k] using the method of partial fraction expansion. 


F(z) z?+zi+1 2+z+1 
0.227 +0.9z14+1 (z+0.4)(z+0.5) 
F(z) 24+ztl k, k, k, 


= + 
Z 2(z+0.4\(z+0.5) z zt+04 2z+0.5 


Calculation of k; 


Sim? @ exosveas 


z7-05 Z 


19z 15z 
z+04 2z+0.5 


Hence: F(z) =5 
So, f[k] becomes: 


fik=Z"[F@ |= (58[k] — 19(-0.4)* + 15(-0.5)*) ufk] 


b. Computation of f(k) using the method of complex integration. 


fines [ F(z) | = >’ residues F(z)z*" => 


S ; (22 +z4+1)z*1 
Uk =) residues (z+0.4\z+0.5) 


For k = 0, the poles are: z = 0, z = -0.4,z = -0.5 


Z+z4+1 


= =5 k= 0 
(z+ 0.4)(z + 0.5) 


residue z=0—73 


2 


0.227 +0.9z14+1 
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2; k-1 
residue 2=-0A—> @ 42412 (760.4) = -19(0.4)" (8) 
z+0.5 
2 k-1 
residue z=-0.5—9 @ 72402 ( = 75-05) = 15(-0.5)! (0) 
z+0.4 
Therefore: 
lk] =(58[k] + 19(-0.4)* + 15(-0.5)") uf k] (10) 
2 
3.7.14 A discrete-time system has a transfer function: H(z) = = 7 
2 
gel. 
4 


Calculate the output, y[n], if the input is the ramp function x[n] = n. 


SOLUTION: Calculation of y[n]: 


yln] = 2 {¥@)} = 2 {H@)X(2)} = ylal= 27 7 ri oa (a) 
4 
Y@) _ 42? 
E 1) (244) 1)° 
2 2 0) 
k, k, ks ks 
= + + + 5 
(zt) (ct) 
2 2 
Calculation of k; 
k= "0, ef =4 
zo, Z 2 
ee "Ol 241). 4 
wl Z 2 9 (3) 
_ d(Y(z) »\_ 32 
ks, tim (1 | 5 
Y 
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Thus: 


By using the inverse z-transform, the output y[n] of the system is computed: 


1)" 4( 1) 32 16 
viot-[4(2) Al *] 218) ©) 


3.7.15 Compute the inverse z-transform of the following functions. 


_ 10z _ wt+z 
B= Ga e-05)' "Goran 
2 +z+2 _ (z+) 


F(z) = F,(2) 


(2-12? =241) ~ (z-1)(z—-.25) 


SOLUTION: 


a F(z) _ k, ‘ k, 7 20 rn 20 (1) 
z z-l1l z-05 z-1 z-0.5 


Thus, 


E(z)=20—--20 *_ = 
z—-1 z—-0.5 (2) 


Alk] = IZT[F,(z)] = 2001 - 0.5" uk] 
b. First suggested solution: partial fraction expansion 


F(z) 227 +1 k, ky kyo 


z @-2)-1) z-1 2-2 (2-2 @) 
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2 i ~1)— (272 = 
ky =| | ~tim| 422-0 Qz +1)|_8-)_, 6) 
z22| z-—l z32 (z _ 1) 1 
3 1 9 
H i 2( ) _ 
a z-1 z-2 (z-2/ ”) 
IZT 
or F,(z)=3 . ae | ae = 
-1 z-2  (z-2) 
filk]=Z7[F,(2)] = (3- 2° + 9k2° uk] (8) 
Second suggested solution: complex integration 
3 2: 
BW 2z = ZL a +1) = 
(z-2)°(z-1)  (z-2)°(z-1) 
k 2 
Fak? = 2 Cz +) ) 
(z—2)°(z-1) 
Using the complex integration method, we get: 
k vs 
residue| => jin 2D —- 3 (10) 
z=1 z1 (z = 2) 1 


residue 
z=2 


k (5,2 42. kV 
aint a(z (22 a) Lyi E +2Z 2 
z>2| dz z-1 z2 z—-1 


ig| 2+ + ke) (2-1) — (22 2). 


(2-1) 


zZ2 


(2¢k+-2)2*7 +k2*} (2.27 424) 
; = 
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9k? 4s p2gk+1 4 k2*k4 me kre a 9k 


= 42k 4.8.2 + Kok _g.nk 9k =(4e+4)o! ak — 2k ok _ ot 
2 2 3 
residue = 9k - fey (11) 
So, folk] = (3+ 9k2*7 — 2* yuk] (12) 
; e 
” F(z) _ Z+z4+2 _ ki k, + k, 4 Kaaks (13) 


z  xz-Wzz—-zt+1) z z-l gage gos 


k= tim( B@,) = n,) (14) 
z0 Zz (-1)1 

_.{ 5) — 14142 _ 
k= bin( a p= tes aa m2) 


—0.5 + j0.866 + 0.5 + j0.866 +2 2 + {1.732 


(0.5 + {0.866 — 1)2 jsin o (-0.5 + j0.866)2j0.866e'? 


2.64540" 


Z-299.1° 
= 260° 4120°4 490°7 739 = 1.53 


ee Osc aad (16) 


148 Digital and Statistical Signal Processing 


Thus: F,(z) = -2+ ar ce a hoe (17) 
— 


jz _f 
Z=e% z=e°° 


k 


4 


k 
falk] = -26[k] + 4u(k) + 1.5307" (5 +1.53¢!4 fee => 
ska kn 
f[k] = -26[k] + 4u(k) + 1.53( "i284" + sraig te | 
1 { ka 
= —26[k]+ 4u[k]+ 1.53( 7-24) ss AE) 


falkl= [-2a18 + 4u[k]+3.06 cos( © = 229.1} (18) 


F(Z) _ (z+1) =: A 7 B " C (19) 
"Zz  (z-1)(z-0.25) ~=z-1 (z-0.25" (z—-0.25) 


d 


Subsequently, another calculation method for A, B, C is provided. 


2 ME) )! 2? 2356 (20) 
(2-025, (0.75) 


1 


To derive B, we proceed to a multiplication with (z — 0.25)? 


as 2 = - (z—0.25)? + B+ C(z— 0.25) 
Z— are 


Therefore, at z = .25, we have 


1.25 5 4 5 
0.75 43 3 


(z+1) 
~ (z-1) 


1.67 (21) 


z=0.25 


Calculation of C 


z-Transform 


Thus, 


F(z)= = + 
(z—1)(z—0.25) z—-1 (z—0.25) (z—0.25) 


-3.56z 


d lanl 1(z—1)-1(z +1) 
C= = 

dz| (z—1) 2=0.25 (z-1) 
c= | = _=-356 

=1) ey (0.75) 

(z+1) _ 3.56z -1.67z 


The second term can be written as: 


Thus, 


z  _ -167  0.25z 


0.25z 


7 — — 
(z-0.25)?  .25 (z—0.25) 


(z— 0.25) 


fikl= [ 3.56 — 6.68 - k(.25)* — 3.56(.25)* | u[k] 


3.7.16 Compute the inverse z-transform of: 


SOLUTION: 


The k; coefficients are calculated by using Heaviside’s formula. 


ky = 


X(2) Zz +6z 
(z?— 22+ 2)(z—1) 
X(Z) _ ky + ky ks 
Zz z-1 z-1+j z-1-j 


2° +6z 


k, =lim—, (z-l=7 
z=1 (z° —2z4+2)(z-1) 
lim z+ 6z és = 1+ j+6 
z1+j (z? — 27 4+2)(z—1) (1+ j-1)(1+j-1+)) 
7+j 7+j 7+ij 7 Jj 
p2p 2p a a: 


149 


(22) 


(23) 


(24) 


150 Digital and Statistical Signal Processing 


bocgtgok 4) 
Hence, 
=7— 7 gg) as 7 1 Zz 
X(z)=7 al 2 i} (2 ig} (5) 
@satcl=7+(—Z+jA](vay eo +(2 jh valet = 
xk] =7-2 (v2) eft 7 Bye 
jh (2) oj La) elt 
st=7~2( da) Le ve] jv) Le 0%" 


3 
: +1 
3.7.17 Compute the inverse z-transform of X(z)= = 


z-—27?-z-2 


SOLUTION: The denominator of the given function is expressed as: 


A(z)=2° —2° -z-2=(z—2)(z+0.5 + j0.866)(z + 0.5— j0.866) 


So, XB 
Z 


is analyzed into a partial fraction sum: 


X(Z) _ Cg Cy a C3 
Zz 


; + (2) 
z Z+0.5+ 0.866 z+0.5—j0.866 z-2 


z-Transform 


The c; coefficients are calculated by using Heaviside’s formula. 


Cy = 20.0] wih = 95 
Zz _, —2 
L z=0 
x F 

6) oxo jo.) = 0.429 + 0.0825 

| Z 2=-0.5-j0.866 
ge) >| = 0.643 

L Zz z=2 


Replacing the corresponding values of Eq. (3), X(z) is calculated, and then, by 


using the inverse z-transform, x[n] is obtained. 


X(z)=Cy+ nie + me - 
z+0.5+ j0.866 z+0.5-j0.866 z-—2 
Sa Cy C C3 


+ + + 
1+0.5+ j0.866z' 14+0.5-j0.866z' 1-2z' 


x[n] = c)d[n] + c,(—0.5 — 70.866)" u[n] + c,(—0.5 + j0.866)" un] + c,2” u[n] 


Based on Eq. (6) and substituting in Eq. (5), x[n] is derived: 


|p| = V(0.5)? + (0.866)? =1 
40.866 _ 4m 
05 3 
icx| = (0.429)? + (0.0825)? = 0.437 
_; 0.0825 
0.429 


Zp,=m+tan ad 


Zc, =tan = 0.19 rad (10.89°) 


x[n] = c)d[n] + c,(-0.5 — 70.866)" u[n] + c,(-0.5 + 70.866)" ulm] + c,2” u[n] 
=C,)d[n] +2 lc,| Ip: | cos(Zp,n+ Zc,)+ c;(2)" un] 


= -—0.56[n] + 0.874 cos (= n+ 03 + 0.643(2)" u[n] 
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This result can be also be verified using MATLAB, by implementing the follow- 


ing code. 

num = [1 0 0 1]; 

den = [1 -1 -1 -2 0]; 

[Ir, p] = residue(num, den) 
tS p= 

0.6429 2.0000 


0.4286 — 0.8251 -0.5000 + 0.8660i 
0.4286 + 0.8251 -0.5000 — 0.86601 
-0.5000 0 


The first 20 output samples are calculated by the following code: 
num = [1 0 0 1); 
den = [1 -1 -1 -2 0]; 


x = filter(num, den, [1 zeros(1,19)]); 
3.7.18 Compute the inverse z-transform of: 


Mes OS). (z -1)(z+0.8) 


cece (z+0.5)(z+ 0.2)’ ie (z= 0.5)°(z +0.2) 
SOLUTION: 
a X(2y = =e + 0.8) 
(z+ 0.5)(z + 0.2) 
X@)_G, G Cs (1) 


+ 
Zz z z-05 z+0.2 
The c; coefficients are calculated using Heaviside’s formula. 


_ X@) _X@) 


a Fhe 8 C= ee 0.5)| 5 = -1.857, 
C; = ese +0.2)|_ =-5.143, 
Zz z=-0.2 
X(2)=8 1.857z 5.143z 


z-05 2402 
x{n] = IZT { X(z)} = 86[n] - 1.857(0.5)" u[n] - 5.143(- 0.2)" uf] 
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(z* -1)(z+0.8) 
(z—0.5)°(z +0.2) 
X(z) _ Cy i Cy 4 C3 si C4 
z z z+0.2 z=-0.5 (z=05) 
= X(z) Ls = -16, C> = X(z) 
Zz = Zz 


=-2.79, 


b. X(z)= 


e (z+0.2)|,__,, =5.88, 


z=-0.2 


C= X@) (,_0.5)| 
: 


=11.12 


C= d ( (z’-1)(z+0.8) 
9 dz z(z + 0.2) ae 


5.88z 11.122 2.792 
z+0.2 z-0.5 (z-0.5) 


x[n] = -166[n] + 5.88(— 0.2)" u[m] + 11.12(0.5)" u[n] — 2.79n(0.5)" u[n]. 


X(z) =-16 


3.7.19 Use the method of analysis in simple fractions and the division method to find 
the inverse z-transforms of the following signals: 


z Tae 1 aa 1 
a. X(z)=——_,__ b. X(Z)= 3 lzl> 5 c X(z)= i lzl> 2 
ep) 1+—z'+—z? Le 


SOLUTION: 
a. X(z)= oF = a . 

(z-1~) 2? +:2z+1 P+ 3+ Y,) 
yn A ch | 
2(1+¥) ae 
“fac pales 

Thus, 
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b. Analysis in simple fractions: 


yi 
X(z)= 2 - 4s = 
d4izt4iz7% 14227 14227 
1a5z 1n5z 
A= 1 La 4 3 and B= 1 eo 4. 
1+—Zz 1+=z 
2 2 


Thus, 


ioe 1 el ee 
2, 4 

1-2 ,7-1.,2 fe gt gees 
4 8 4 16 

3,11 

4 8 

ee 

4 16 32 

Le ae 

16 32 


For n = 1, from the relation x[n] -| a i) mi 1) ea we get: 


1 1 3 5 
x[1] = [-a(—1}+4(-2)}wnt-+3-2- “V which is in agreement with the 


corresponding division coefficient. 
c. Analysis in simple fractions: 


z-Transform 
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3.7.20 Solve the following difference equation, where the initial values are zero. 
y[k +2]-0.3y[k + 1]+ 0.02y[k] = (0.01)(0.3)* ufk] 
SOLUTION: We apply the z-transform in the given difference equation: 


27Y(z)—0.32Y (2) + 0.02Y(z) = 0.01- a (1) 


(1) > Y(z)(z2— 0.32z+ 0.02) = 0.01 5 
z-0.3 


0.01z 
as (z—0.1)(z— 0.2)(z — 0.3) @) 
Yz)_ OA B C 


z (2-01) (@-02) (2-03) 8) 
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2 0.01 ae! oe 

~ (z-0.2)(z-0.3)|,_,,  (-0.1(-0.2) 

_ 0.01 — 0.01 | 

~ (z-0.1)(z-0.3)|,_,,  (0.1)(-0.1) 

_ 0.01 — 0.01 _ 

~ (z-0.1(z-0.2),_,, (0.20.1) (4) 
Thus, 

ylk]= (0.5(0.1)' =(0.2)* + 0.5(0.3)* )ufk] (5) 


We can now check the solution. From the given difference equation, the first 
non-zero terms are: 


k =0:y[2] - 0.3y[1] + 0.02y[0] = (0.01), thus, y[2] = 0.001 


k=1: y[3] - 0.30.01) = (0.01)(0.3), thus, y[3] = 0.006 


From the analytical solution of Expression (5), we have: 


y[0] = (0.5-140.5)=0 
y[1] = (0.5(0.1) - (0.2) + 0.5(0.3)) = (0.05- 2 +0.15) = 0 


y[2] = (0.5(0.01) — (0.04) + 0.5(0.09)) 
= 0.005 — 0.04 + .045 = 0.001 


y[3] = (0.5(0.1)3 — (0.2) + 0.5(0.3)°) 
= ,0005 —.008 + .0135 = 0.006 


The derived values agree with the previously derived solution; so, the result is 
correct. 


3.7.21 Solve the second-order difference equation y[k + 2] — 5y[k + 1] + 6y[k] = 0 with the 
initial conditions y[0] = 0, y[1] = 2. 


SOLUTION: We apply the z-transform in the given equation, thus: 
Z| y[k + 2]—5Sy[k +1] + 6y[k]]=0 


= 2Y(z)—z’y[0] - zy[1]-5(z-Y(z)—z-y[0]) + 6Y(z) =0 
=> 2’Y(z)-2z—-5zY(z)+6Y(z)=0>5 
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Y (2\(2? -5z+6)=22 > Y(z)= (1) 
z—5z+6 
It is: 

YZ) 2 2 _ kk 0) 

z  2-5z+6 (z-2\(z—-3) z-2 2-3 
k= lim” (2—2)=-2 @) 
atime eae2 (4) 

z23 «7% 
Thus, 


y(e)=2| *@~ 2z mn 2z = 
Zz z—-2 z-3 (5) 


ylk] = IZTLY (2) =(-2(2)' + 203)" uk] 
3.7.22 Solve the second-order difference equation: y[k + 2] + 5y[k + 1] + 6 + y[k] = 5u[k] 
with the initial conditions: y[0]= -12, y[1] = 59. 
SOLUTION: We apply the z-transform in the given equation and then calculate a : 
Zz 
2Y(z)— 2? ylo]- zyl1]+5[ zY(z)— zyl0]]+ 6Y(z) = er 
ey — 

2Y(z)— 27(-12) — 2(59) + 5zY (z) — 5z(-12) + 6Y(z) = a 

Zz — 


(22 +5z+6)Y(z) = 122? $at5 
Zz 


1227 +z m 1 5z 


Y(z)= : 
(2) 245246 27+5z+6 z-1 


Y(z)_ (12z+1)(z-1)+5_ K, K, rn K, 
Zz (z+ 2)(z+3\(z-1)  z+2 z+3 z-l1 


(1) 
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We calculate the corresponding numerators: 


gases # ES DS 40) _ 64 

(z+ 2)(z+3)(z-1) La, 
oe (12z+1)(z-1)+5 (z+3) _ 7135 

(z+2)(z+3)(z-1) = 4 (2) 
a ee 1) _5 

(z+ 2)(z+3)(z-1) a 


We substitute the values of K, into xe and then we multiply with z, hence: 
Z 
¥@= 64/3z + 135/4z 5/12z (3) 
zZ+2 zZ+3 z-1 


By using the inverse z-transform, we get: 


64, 135, ar, 5 
k]= 2 5) k 4 
wes=] poser eee (4) 
3.7.23 Let the following equation describe a discrete-time system: 


y[k] = 0.3y[n —1]+0.7x[n] and n 20 


a. Compute the frequency response, H(e/Q) 
b. Compute the response of the system when the input is: x[n] = sin(0.32n) for n > 0. 


SOLUTION: We apply the z-transform to the given equation: 
y[n] = 0.3y[n-1]+ 0.7x{n] > Y(z) = 0.3z'Y(z) + 0.7X(z) 
= (1-0.3z27)Y(z) = 0.7X(z) 
The transfer function will be: 


_Y¥Z)_ 07 | O7z 


H@)= y= aa 
(z) 1-0.3z z—0.3 


The frequency response is calculated by substituting z = e/®: 


_ Ae 0.7e™ (2) 
ze 6/9 _ 93 cosQ-0.3+4 jsinQ 


H(e!) = H(z) 
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The magnitude of H(e®) is 


: jw 
He) _ 0.7e == 0.7 (3) 
cosw—0.3+ jsinw] |(cosw — 0.3)? + sin? w 
and the phase of H(e®) is 
(Q)= Ar Wa <7" (4) 
e § cos Q—0.3+ jsinQ cosQ—0.3 


b. For an input signal x[n] = sin (0.3 xn) when n > 0, it is: Q = 0.32 = 0.94rad 


For Q = 0.3 1, the magnitude of the frequency response is |H(e/)| = 0.815, and 
the phase is @Q) = 0.34 - tan1(2.81) = -0.297ad. 


The response of the system is y[7] =e!" -0.815e7/°”? = 0.815 -e/0°" 9) 


3.7.24 Derive the inverse z-transform of X(z)= = with ROC |z| > 1. Study the 
stability of the system. a E41 


SOLUTION: From the form of the function and the transforms (10) and (13) of the Table 
(sin Q)z (1) 


3.1, we choose to use: (sinQn)u[n] — — 
z —(2cosQ)z+1 


Comparing the denominators yields: 


3 


2cosQ=1=> cosQ=0,5> cosQ=0,5 or Q=—. Then, aa a and the 


a 
3 


nominator can be written as: 3z = ° : sin( 


X(z)= 


»( 5) co 2sBsin( % tn (2) 


3 
ae 2 acos{#) e+ 


An LTI system that has a rational transfer function (with regards to z), is casual 
when the ROC is in the exterior of the circle, defined by the most remote pole and 
the rank of the numerator is less than or equal to the rank of the denominator. 


The system under discussion is casual. 


We can decide on the stability of a causal ee. by ae the position of 


the poles in the unit cycle. The poles are z, =e is and z, =e ts . The magnitude of 
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the poles is equal to 1, so the poles are located on the unit circle, and therefore the 
system is marginally stable. 


3.7.25 Study the stability of the following system: 


(=z) , 
X(z)= ———~ with ROC|z| > 0.4. 
(z>—0.4z + 0.16) 


SOLUTION: Using the criteria of stability of the casual LTT systems referred to in the 
solution of Problem 3.7.24, we study the position of the poles with regards to the unit 


circle: The poles of the system are Z, = 0.4e'3 and z,=0.4e 3. The magnitude of the 
poles is 0.4. Consequently, the poles are positioned inside the unit circle, and the 
system is stable. 


The stability of the system could also be decided by the fact that the unit circle 


belongs to the ROC. 
z,=0.4e'3 =0.4| cos| ~ | + jsin|  |]=0.4 1, x3 = 0.2 + j0.348 
3 3 a * 3 
Zy= 0.4e'3 = 04) col) + js 5) 7 ol ts 8 = 0.2 — j0.348 


3.7.26 Discuss the stability of the LTI systems described by the following difference 
equations. 


yln]—+y[n 1+ yln 2]= x{n] 


2 


3 1 
yln]——y[n-1]+ —y[n- 2] = x[n] 
a 8 
SOLUTION: a. By z-transforming the first difference equation, we have: 
34 3» 
Y(z)- a Y(z)+ 4 Y(z)=X(z)=> 


[a-$e" deve =X(z)> 


He)= 1 3 
(z) {= lg ag 
2 4 
42? 
fey 
@) 42? —274+3 


2+ /44 


8 


A=4-4.4.3=-44, 2) = =0.2+ j0.8 
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The poles are included in the unit circle; therefore, the system is stable. 
b. By z-transforming the first difference equation, we have: 


Y(z)- 52 Ye) + =27Y(2) = X(z)> 


(1-3 z 145 27} ¥@)=x@)> 


The system is stable because it is casual and the poles are included in the unit 
circle. 


3.7.27 Compute the output, y(1), of a system with the impulse response hi[n] = (2) un], 
when the input signal is: 3 


a. The complex exponential sequence: x[n] =Ae™”? 
b. The complex exponential sequence: x[n] =Aei™" 


c x[n]= 10—5sin n+ 20cosan 


SOLUTION: It is well-known that when an input of frequency, @, is applied to a linear 
system, the output is of the same frequency but of a different amplitude and phase. 
The new amplitude and phase are defined by the frequency response of the linear 
system at that frequency. 
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The frequency response in MATLAB can be calculated as follows: 


= [0:1:500]*pi/500;% 501 frequencies in the interval 0 - 1 


= 1; % vector of coefficients for numerator of H(z) 


Ww 
a 
b = [—1, -2/3]; % vector of coefficients for denominator of H(z) 
H 


= freqz(a,b,w); scalculates frequency response for frequencies 
in vector w 


plot(w/pi, abs(H))% abs returns the magnitude of complex value 
xlabel('w/pi for O<w<pi') 

ylabel('Magnitude |H(exp(jw)|') 

title('Frequency response of H(exp(jw))') 


Frequency response of H(exp(jw)) 


N 
a 


1.5) 


Magnitude |H(exp(jw)| 


0.5 1 1 1 
0 0.2 0.4 0.6 0.8 1 


w/tt for O<w<r 


We notice that |H(0)| = 3. In the case that the frequency response has to be 
designed in the interval -x < w < x, the following change has to be made in the 
MATLAB code: 


w=[0:1:500]*2* pi/500;% w from 0 to 2*pi 
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a. Since the input x[n] = Ae’"” is of the form Ae/" with frequency w = 1/2, then: 
y[n]=H (ei/2) Agim? 


1 1 
H(e™) = == = 
1-feh? 1448) 


Thus, 


3 m33,7°) 


ln] = 2 Ael 


V5 


b. Similarly, the second discrete signal is of a frequency z. 


So, 


Thus, 


y[n] = = Aci 


c. The third signal has the following frequencies: 0 (constant), 1/2 and x. The 
response of the last two frequencies has already been calculated. So, for w = 0: 


1 1 cae 


H(e”)= = 
ey 1-fe" 1-2 1 


Thus, 


un) =3-10-3.5-sin{ 33.6 }+ 20. coscem 


5 2 V5 


Validity of the output of the system for w = 0: h[n] is produced by the difference 
equation y[n] = (2/3) y[n — 1] + x[n]. Therefore, if we consider the case of a constant 
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input x[n] = 10, n = 0, 1, 2, ... that has w=0, the output produced by the aforemen- 
tioned difference equation is: 


n=0  y[0] = 10 


n=1 _ y[l] = (2/3)*10 + 10 = 16.667 

n=2_ y([2] = (2/3)*16.67 + 10 = 11.11 + 10 = 21.11 
n=3_ y[3] = (2/3)*21.11 + 10 = 14.07 + 10 = 24.07 
n=4__ y[4] = (2/3)*24.07 + 10 = 16.05 + 10 = 26.05 
n=5 _ y([5] = (2/3)*26.05 + 10 = 17.36 + 10 = 27.36 
n=6 y[6] = (2/3)*27.36 + 10 = 18.24 + 10 = 28.24 
n=7__ y[7] = (2/3)*28.24 + 10 = 18.82 + 10 = 28.82 
n=8 _ y([8] = (2/3)*28.82 + 10 = 19.22 + 10 = 29.22 
n=9 — y[9] = (2/3)*29.22 + 10 = 19.47 + 10 = 29.47 
n=10  y[10] = (2/3)*29.47 + 10 = 19.65 + 10 = 29.65 
n=11_ y[11] = (2/3)*29.65 + 10 = 19.77 + 10 = 29. 


We can see that, after some time (transient time), the output, at its steady state, 
takes the value 30, just as we previously calculated using the frequency response 
of the filter. 


3.7.28 Let the inverse z-transform: 


1+2z7 


X(z)= 
@) 14+0.4z1-0.12z7 


Use the impz.m and filter.m commands in MATLAB to calculate the first 11 
samples of the x[n] sequence. 


SOLUTION: 

a. MATLAB code using impz.m command: 
L = 11; % Length of output vector 
num = [1 2]; 
den = [1 0.4 -0.12]; 
ly, t] = impz(num,den,L); 
disp('Coefficients of the power series expansion'); 
disp(y') 

b. MATLAB code using filter.m command: 
N = 11; 
num = [1 2]; 
den = [1 0.4 -0.12]; 
x =[1 zeros(1,N-1)]; 
y = filter(num,den, x); 
disp('Coefficients of the power series expansion'); 
disp(y) 
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The result for both programs is the same: 


x{n]= {1.0000 1.6000 -0.5200 0.4000 -0.2224 0.1370 
-0.0815 0.0490 -0.0294 0.0176 -0.0106} 


3.7.29 Compute the z-transform of the function x[n] = [3 5 4 3],0 <n <3 using MATLAB. 


SOLUTION: First suggested method 
syms z 
xX0=3; xX1=5; x2=4; x3=3; 


Xz=x0*(z*0)4+x1*(z*-1) +x2*(z*-2)4+x3*(z*-3) 


pretty (X) 
One 
Zoe 2 
Second suggested method 
syms Zz 
x=[3 5 4 3]; 
n=[0 1 2 3]; 
X=sum(x.*(z.*-n)) 
pretty (X) 
X@)=34 24 S4 
ZF 2 


3.7.30 Compute the z-transform of f[n] = 2” using MATLAB. 


SOLUTION: The code in MATLAB is: 
syms n Zz 
f= 2*n ; 
ztrans(f) 
simplify(ans) 


F(z) =z/(z-2) 


z 
To verify the above result, we calculate the inverse z-transform of F(z) = = 


syms n Zz 
F=z/(z-2); 
iztrans(F) 


ans =2*n 
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3.7.31 Compute the z-transform of the following sequences using MATLAB: 


O[n],u[n],n-uln],a"u[n],na"u[n]cos(@yn)u[n], sin(@yn)u[n], 


a" cos(@ n)u[n],a" sin(@on)u[n] 


SOLUTION: In the following Table 3.5, the main transform relations are provided, 
while their accuracy is verified with the aid of ztrans and iztrans functions. 


It is noteworthy that the ztrans function in MATLAB calculates the single-sided 
z-transform (n > 0), so the unit-step function can be omitted. 


3.7.32 Rewrite the following function in a partial fraction expansion formulation. 


27 4+3z4+1 
tga 
2) 234577 +2z-8 


TABLE 3.5 


z-Transforms and Corresponding MATLAB Functions 


Discrete-Time 


Domain n z-Domain Commands - Result 
x[n] X(z) syms n Z aw 
6[n] 1 f=dirac(n) ; ans = 
ztrans (f£,z) dirac (0) 
= O00) ed 
u[n] z/(z-1) f=heaviside (n) ans = 
ztrans (f£,z) z/(z-1) 
n-u[n] ztrans (n,2z) ans = 
Z as 
ra z/(z-1)*2 
(z-1) 
au[n] . F=z/(z-a); f = 
f=iztrans(F,n) a“n 
zZ-a 
na"u[n] f=n*a“n; ans = 


cos(@)n)u[n] 


sin(@ n)u[n] 


a"cos(a@pn)u[n] 


a"sin(a@on)u[n] 


az 
(z—a)" 


2? —zcos(@,) 
2? —2zcos(@))+1 


Zsin(@)) 
2? —2zcos(@))+1 


2? — azcos(@) 


2 


2? —2azcos(@)) +a 


az sin(@,) 


2? —2azcos(@)) +a” 


ztrans(f,z) 


f=cos (w*n) 
ztrans(f,z) 


f=sin(a*n) ; 
ztrans(f,z) 


f=(a*n) *cos (a*n) 
ztrans(f,z) 
simplify (ans) 


f=(a*n) *sin(a*n) ; 
ztrans(f,z) 
simplify (ans) 


z*a/(-z+a)*2 


ans = 

(-z+cos(w))*z / 
(-2*24+2*z *cos (w) -1) 
ans = 

z*sin(a)/ 
(z*2-2*z*cos (a) +1) 
ans = 
-(-z+cos(a)*a) *z/ 
(z*2-2*Z * 

cos (a) *a+a*2) 

ans = 

z*sin(a)*a/ (2*2- 
2*z * cos(a) *ata*2) 
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SOLUTION: 
% Calculation of denominator’s roots 
A=[1 5 2 -8]; 
riz=roots(A); 
% Calculation of numerators of the partial fractions 
syms z 
X=(Z*24+3*Z4+1) /(2*34+5*Z*24+2*z-8); 
cl=limit ((z-riz(1))*X,z,riz(1)) 
c2=limit( (z-riz(2))*X,z,riz(2)) 


e3=limit( (z-riz(3))*X,z,riz(3)) 
Hence, we get: 


27 4+3z4+1 1/2 1/6 1/3 
X(z)= p ; = 
z+5z°4+2z-8 zt+4 z+2 z-1 


3.7.33 Rewrite the following function in a partial fraction expansion formulation. 


_ 2 +3z41 


X(z)= 
(@) z-—3z4+2 


SOLUTION: First suggested solution 
% Calculation of denominator’s roots 
A=[1 0 -3 2]; 
riz=roots (A) 
riz = -2.0000 1.0000 1.0000 


Observe the existence of a double root at point 1.0000 
X(z) can be expressed as: 


Cy Cy C, Cr+ Cy 
a eer 


a greet ; 
Z— Pp, (z— p,) (Z- pr) Z— Presi Z— Py 


X(z)= 


The coefficients c,, ..., Cc, can be calculated as: 


r-1 _—7.\t 
c; = lim a d ((s Pi) X@) 1, 
24; (r—1)! az 


eee ep) 2) pi=rtl..n 
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2 


% Calculation of c, 

syms Z 

X=(z*24+3*Z4+1)/(z*3-3*z+2); 

cl=limit ((z-riz(1))*X,z,riz(1)) 

% Calculation of c, (i=1) - 2 common roots (r=2) 
r=2 

% definition of (z — p,)*X(z) 
f= ((z-1)*r )*xX; 


a’ ((z—p,)'X(2) 
dz" 


° 


% Definition 


par=diff(f£,z,r-1); 


% Calculation 


r—i)! 
fact=1/factorial(r-1); 
% Calculation c, 
c2=limit (fact*par,z,1) 
% Calculation c, (i=2) 
par=diff(f,z,r-2); 
fact=1/factorial(r-2); 


limit (fact*par,z,1) 
Therefore: 


2 +8ee) _ 1/9 10/9 5/3 


xX = 
®) 2-3z+2 z+2 z-1 (z-1¥ 


Second suggested solution 


X(z) can be converted to partial fraction expansion formulation with the use of 
the residue command. 
% Define the coefficients of numerator and denominator 
num=[ 1 3 1]; 
den=[ 1 0 -3 2] 


% Use of residue command 


[R,P,K]=residue(num,den) 
X(z) can now be expressed as a fractional expansion: 


V9 109, 5B 
z+2 z-1 (z-17 


X(z)= 


which is the same derived previously. 
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3.7.34 a. Solve the following difference equations using the z-transform: y[n] — y[n — 1] = 
x[n] + x[n - 1] where x[n] = 0.8". 
b. Plot the results for 0 <n < 20. 


c. Verify the accuracy of the solution by replacing the result in the given differ- 
ence equation. 


SOLUTION: a. The code in MATLAB is: 
syms n z Y 
x=0.8%n; 
X=ztrans(x,z); X1l=z*(-1)*X; 
Y1=z~*(-1)*Y; 
G=Y-Y1-X-X1; 
SOL=solve(G,Y); 
y=iztrans(SOL,n) 


Then y[n] is given by: 


y=10-9*(4/5), 


b. The code in MATLAB is: 
n_ s=0:30; 

y _s=subs(y,n,n _ s); 
stem(n__s,y_ 8); 
legend ('SOLUTION') 


10 


J u PY YYYY YT ¥ 
8 L 
6 L 
4 LL 
| 
al 
0 5 10 1 2 25 30 


c. The code in MATLAB is: 
xn=0.8*(n); 
xn 1=.8*(n-1); 
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yn=10-9*(4/5)“n; 

yn _ 1=10-9*(4/5)*(n-1); 

test=yn-yn _ 1-xn-xn_ 1 

test =8*(4/5)*(n - 1) - 10*(4/5)*n 
simplify(test) 


ans =0 


We would expect to get: test = 0. By using the simplify command, we simplify 
the answer, so we got: ans=0, hence the answer is right. 


3.7.35 Let a casual discrete-time system be described by the following difference equa- 
tion y[n] = 0.9y[n — 1] + x[n]. For this system plot using MATLAB: 


a. the magnitude of the frequency response, and 
b. the zeros and poles 


SOLUTION: a. We z-transform the given deference equation: 


y[n] = 0.9y[n-1] +x[n] > Y(z)=0.9z TY(z)+ X(z) > H(z) 


Y(z) 1 \2| 
= =—_—_——_ _|z/>09 
X(z) 1-0.9z7 
The magnitude of the frequency response is: lH (e!”)| = = 
e” —0.9 


We use the freqz command in MATLAB to design |H| in the frequency interval 
@ =0- «rad: 


a=1; 

b=[1, -0.9]; 

[H,w] =freqz(a,b,100); 
magH=abs(H); 

plot(w/pi, magH) 
xlabel('Frequency in pi units') 
ylabel('Magnitude') 
title('Magnitude Response') 
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Magnitude Response 


10 


Magnitude 


0 0.2 0.4 0.6 0.8 1 
Frequency in 7 units 


With the following code, we design the frequency response magnitude for the 

frequency range 
o=0-2n rad, and we notice that the figure is symmetrical. 
a=1; 

b=[1, -0.9]; 

[H,w]=freqz(a,b,200, 'whole'); 

magH=abs(H); 

plot(w/pi, magH) 

xlabel('Frequency in pi units') 
ylabel('Magnitude') 

title('Magnitude Response') 


Magnitude Response 


10 


Magnitude 


Frequency in 7 units 


172 


Digital and Statistical Signal Processing 


In the code that follows, we design the magnitude of the frequency response for 
the frequency range w= -2- x rad. 


a=1; 
b=[1, -0.9]; 

w=[-pi:pi/200:pi]l; % which is w=[-200:1:200]*pi/200; 
H=freqz(a,b,w); 

plot(w/pi, abs(H)) 

ylabel('Magnitude') 

xlabel('Frequency in pi units') 

title('Magnitude Response’) 


Magnitude Response 


Magnitude 


Frequency in 7 units 


b. We plot the poles and zeros by using the zplane command as shown below: 


a=[1]; 
b=[1, -0.9]; 
zplane(a,b) 
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1 | 

© 05} 
Oo 
eo 

© @) r O x 4 
D 
oO 

£05} | 

oq f | 


-1 -0.5 0 0.5 1 
Real Part 


It is easily understood that there is a root in the numerator at z = 0 (a zero for 
H(z)) and a root in the denominator at z = 0.9 (a pole for H(2). 


3.7.36 Plot the magnitude of the frequency response of the system with impulse 
response 


a. For the range w=0-x and 


b. For the range w=-1-n. 


SOLUTION: The solution is: 


=el® 
ze 1 


nod=(2] u[n] i) H@)=—3—5 i H(e”)=——4 se 


a. We write the following MATLAB code: 
w=[0:1:500]*pi/500; % 501 samples in the range 0 - a 
a=1; % coefficients of the numerator of H (z) 
b=[1, -2/3]; % coefficients of the denominator H(z) 
H=freqz(a,b,w); % Calculate Frequency Response 
plot(w/pi, abs(H)) % Plot the magnitude 
xlabel('w/pi for O<w<pi') 
ylabel('Magnitude |H(exp(jw)|') 


title('Frequency response of H(exp(jw))') 
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Frequency response of H(exp(jw)) 
T T T T T 


Magnitude |H(exp(jw)| 
6 
T T T 


= 
T 


= 
a 


| 
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 
w/n for 0<w<n 


b. In order to plot the frequency response at —1 < w < 2, we will only change the com- 
mand to: 
w=[0:1:500]*2*pi/500; % w from 0 to 2*pi 


Frequency response of H(exp(jw)) 


N 
a 


Magnitude |H(exp(jw)| 
on 


= 
T 


o 
a 


0.5 1 15 2 
w/tt for O<w<r 


j=) 


3.7.37 Study the stability of the systems with the following transfer functions. 


_ (2-05) _ (+1) 
AN eogsy e098) 
a Hee! ad pee ee) 


~ (z24.0.5z—0.5)’ ~ "(22 4240.75) 


Write the MATLAB code to plot the step responses of the systems. 
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SOLUTION: 


_ (z-0.5) 


a. H(z)= (2+0.75) 


The system is stable because the pole of the system p = -0.75 is included in the 
unit circle. 


(2? +1) 


Bb. A@)= 005) 


The system is stable because the poles of the system p, = —0.5 and p, = 0.5 are 
included in the unit circle. The zero-pole plots for the above two systems are: 


(a) 


«Bes z(z—1) = z(z—1) 
(z7+0.5z—-0.5) (z-1)(z-0.5) 


The system is marginally stable because one of the poles (p = —1) is on the cir- 
cumference of the unit circle. 


(z2+z+0.75)  (z+0.5+07/)\(z+0.5—07)) 


a. Bee (z-0.5)(z + 0.5) (z-0.5)(z + 0.5) 


The system is stable because its complex conjugate poles p, = 0.86e/126° and 
Pz = 0.86e- 126° are included in the unit circle. The zero-pole plots for the above 
two systems are: 


(d) 
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In order to plot the step responses we write the following commands in MATLAB: 
step([1, -0.5],[1, 0.75]) % for the system a. 


Step Response 


Amplitude 


Time (seconds) 


step([1, 0, 1],[1, 0, -0.25]) % for system b 


Step Response 


3 x 10% 


0 20 4 0 100 120 


0 60 8 
Time (seconds) 
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step([1, -1, O],[1, 0.5, -0.5]) % for the system c 


x 1028 Step Response 


= 
a 


Amplitude 
nm 


-2.5 


0 20 40 60 80 100 120 
Time (Seconds) 


step([1, -0.25],[1, 1, 0.75]) % for the system d. 


Step Response 
0.5 1 


Amplitude 


5 10 
Time (seconds) 


Taylor & Francis 
Taylor & Francis Group 


http://taylorandfrancis.com 


4. 


Structures for the Realization 
of Discrete-Time Systems 


4.1 Introduction 


The implementation of discrete-time systems can be achieved in two ways: 


1. using software, i.e., by programming of either a general or special purpose com- 
puter system 


2. using hardware, ie., in a circuit-like form 


The way we choose to describe a system often determines the required memory 
space, the speed, the quantization error of a digital system, and therefore it constitutes 
a subject of further study. In this chapter, the schematic way (i.e., using diagrams) of 
describing the discrete-time systems is discussed that facilitate circuit analysis and 
implementation. 

In general, the implementation of systems requires storage capabilities of previous 
input-output values of the system and of the intermediate sets of values created (delay 
elements), as well as the ability to use multipliers and adders. The difference equation 
of a discrete-time system (filter) could be interpreted as an algorithm for calculating the 
output sequence, y[n], of the system when the input sequence, x[n], is known. For each 
difference equation, a block diagram can be designed consisting of delay circuits, multipli- 
ers and adders. This block diagram is usually described as the realization structure of the 
system. 


4.2 Block Diagrams 


The block diagram is a graph with nodes and branches constituting its basic structural com- 
ponents. The block diagrams of the basic components, that are being used to implement 
the difference equation of a discrete system, are illustrated in Figure 4.1 and are the adder, 
the multiplier and the delay components. 

The symbol z* represents the delay of one sampling interval (unit delay). 
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y2[n] 
Adder 
yi[n] yi[n]+y2[n] 
a Multiplier 
y[n] ay[n] 
Delay element 
y[n] y[n-1] 


FIGURE 4.1 
Basic implementation components. 


For example, let the difference equation describing a digital filter (system): y [n] = 2y[n—1] + 
3y[n-2] + 5x[n]. In Figure 4.2, the block diagram, which depicts the realization of the differ- 
ence equation, is presented. 

z1 corresponds to one unit delay. In digital implementation, each unit delay can be 
implemented by providing a storage register. If the required number of samples of delay is 
D>1, then the corresponding system function is of z-? degree. 


y[n] 


FIGURE 4.2 
Realization of difference equation. 
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4.3 Realization Structures 


There are four implementation structures: 


1. Direct form I (Figure 4.3) 

2. Direct form II —- Canonical form (Figure 4.4) 
3. Cascade form (Figure 4.5) 

4. Parallel form (Figure 4.6) 


The different structures of implementation have different characteristics with respect 
to the required computational complexity, memory and due to the finite word length effects. 
The system parameters must necessarily be represented with finite precision. The results 
derived from the calculation procedure of each output sample of the system must be 
rounded or truncated at the appropriate finite word length (corresponding accuracy) 
imposed by the computer or circuit used for the implementation. 

Difference equations for discrete-time systems of degree N>2 are usually implemented 
as combinations of first- or second-order structures in series or in parallel to reduce calcu- 
lation errors in their implementation by digital systems (which already use finite precision 
to represent coefficients and calculate operations). 


X[n] 


X[n-1] 


X[n-2] 


FIGURE 4.3 
Direct form I. 
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FIGURE 4.4 
Direct form II or canonical form. 


o a 0 
FIGURE 4.5 


Cascade form. 


X[n] 


Y[n] 


FIGURE 4.6 
Parallel form. 
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The problems that need to be addressed in a series implementation are: 


e The way in which the numerators should be combined with the denominators 
after factorizing the transfer function 
e The connection order of the individual blocks 


e The reduction of the signal amplitude at intermediate points of the structure so 
that it is neither too large nor too small. 


In the case of parallel form, the interconnection order of the individual structures is not 
significant. In addition, scaling is easier, since it can be done for each structure indepen- 
dently. However, the zeros of the individual structures are more sensitive to the quantiza- 
tion errors of coefficients. 


4.3.1 Implementation Structures of IIR Discrete Systems 


Let us consider the case of implementing an IIR discrete system. The general difference 
equation is: 


N bh M 
y{n] = - Dap, ve" -~k] a — ml] (4.1) 
or ¥ byln—k] = oe, x0 mi] (4.2) 
k=0 m= 
For by= 1 we get: 
yin] =} or,x[n—m] +)" byyln—K] (4.3) 
m=0 k=1 


Based on the above difference equation we can implement the discrete system as shown 
in Figure 4.7 in direct form I, which is called so since it can be designed directly from the 
transfer function of the system: 


ae 
H(z) = XO = 10 (4.4) 
bz * 
2 ‘ 
or Yo>, b,z* = xo» Oyz-™ (4.5) 
k=0 m=0 


In Equation 4.5, the terms of the form Y(z)z*‘ correspond to the inverse z-transform of 
y[n — kK]. 
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X{n] Y{(n] 


FIGURE 4.7 
Implementation of the IIR filter in direct form I. 


Figure 4.7 shows the connection of two partial systems in which one implements the 
zeros (i.e., the roots of the nominator), and the other implements the poles (i.e., the roots of 
the denominator) of the transfer function of the discrete system. Because in a discrete LT] 
system, the total transfer function does not change if the order of the individual systems is 
reversed, we plot the equivalent system of Figure 4.8 (for N = M). 

The two branches with the z delays have the same input as shown in Figure 4.8. Thus, 
they are replaced by a single branch as shown in Figure 4.9, which constitutes the imple- 
mentation as a direct form II or canonical form. 


X{n] Y[n] 


FIGURE 4.8 
Implementation of the IIR filter — intermediate stage. 
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X[n] 


FIGURE 4.9 
Implementation of the IIR filter in canonical form. 


Another form of IIR filter implementation is the cascade form, which is produced if we 
express H(z) as presented in Equation 4.6: 


k 
H@ = oy = %] [HO 4.6) 


where H(z) is a first-degree polynomial with respect to z 


1+ az" 
H(z) = ———"“— 
i(Z) oe (4.7) 
or a second-degree polynomial 
1 $+ az +027 
H,(z) = (4.8) 


-1 2 
1 + bz +b,,z 


N+1 
with k the integer part of (N#*) 


The discrete-time system described by Equation 4.6 is realized using a cascade connec- 
tion of k individual systems (Figure 4.5). Moreover, each of them is realized using a direct 
form I or II. 

A fourth version of implementation of an ITR system is the parallel form. In this case, H(z) 
is recorded in the form of: 


k 
H(z) =C + STH) (4.9) 
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where H,(z) is a first-degree polynomial with respect to z 


A: 
A(z) = tH 4.10 
Oe ae bz (4.10) 
or a second-degree polynomial 
-l1 
Oy; + 0442 
H,@) = of F Oyj 411 
@) 1 + bz) +b,,27 el 
N+1 
with k the integer part of (N) and C= ae 
N 


The implementation of the discrete system of Equation 4.9 is accomplished by parallel 
connection of k individual systems (Figure 4.6). Each of them is implemented using direct 
form I or II. 


4.3.2 Implementation Structures of FIR Discrete Systems 


In FIR systems the discrete impulse response is of finite duration and the function H(z) is 
usually written in the form: 


H(z) = S' itn)-2" (4.12) 


n=0 


The function h(n) consists of N samples. Consequently, H(z) is a polynomial of N - 1 
degree with respect to z'. Thus, H(z) has N - 1 poles at z = 0 and N - 1 zeros. FIR systems 
are characterized by non-recursive difference equations that are in the form of: 


yl] = y hlk]x[n—K] (4.13) 
k=0 
or y[n] = ALO] x[n] + hf] x[n—1] + ... + A[N-1] x[N-1-k] (4.14) 


Figure 4.10 illustrates the realization of Equation 4.14 using direct form I. 


x(n] 


h[N-1] 


FIGURE 4.10 
Implementation of FIR filter in direct form I. 
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In case of the cascade implementation, H(z) can be written in the form of a product of 
first- and second-degree terms. 


k 
H(z) = [[4.© (4.15) 
n=1 
where the first-degree terms are: 
H,,(z) = On + Oh .Z (4.16) 
and the second-degree terms are: 
H,,(Z) = Opn +Oy,Z + OlyyZ (4.17) 


with k the integer part of (N#1) 
Equation 4.15 is equal to Equation 4.17, where, if N is an even number, one of the coef- 
ficient, a, will be zero. 


(4) 
H(z) = [ Jo. + OipZ +O yZ (4.18) 
n=1 


Figure 4.11 illustrates the realization of the cascade form where each subsystem is in 
direct form I. 


X{[n] Qlo2 Loww/2) vin] 
> 


QL1(N/2) 


Q2(Nn/2) 


FIGURE 4.11 
Block diagram in cascade form for a FIR filter. 
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4.4 Signal Flow Graphs 


Signal flow graphs (SFGs), like block diagrams, provide an alternative method for graphi- 
cally describing a system. SFG theory was introduced by Samuel J. Mason, and it can be 
applied to any system without having to simplify the block diagram, which sometimes 
can be quite a difficult process. An SFG is actually a simplified version of a block diagram. 
It consists of nodes, branches, and loops. Each node represents a variable (or signal) and 
belongs to one of the following categories: 


a. Source or input node: The node from which one or more branches start and no 
branch ends (Source node - Figure 4.12 (a)). 


b. Sink node: The node in which one or more branches end and from which no branch 
starts (Sink node - Figure 4.12 (b)). 


c. Mixed node: The node that branches can both come in and come out (Mixed node- 
Figure 4.12 (c)). 


A branch connects two nodes and is characterized by its direction and gain. The direc- 
tion shows the signal flow, while the gain is a coefficient (corresponding to a transfer 
function), which relates the variables x, and x,. For Figure 4.13, the relationship x, = ax, 
is true. 

Path is a succession of branches of the same direction (e.g., x, X,, x; and x, in Figure 4.14). 

Forward path is the path that starts from the input node and ends at the output node (e.g., 
X41, Xo, Xz, Xy Xs in Figure 4.14). 

Loop is a closed path that begins and ends at the same node (e.g, x5, X3, X, in Figure 4.14). 
Two loops of a SFG are called non-touching when they have no common node. 


x x 
x X; x x 
x. 
x a b 


FIGURE 4.12 
(a, b, c) Nodes of system flow diagrams. 


FIGURE 4.13 
Branch of system flow diagrams. 
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Loop 


FIGURE 4.14 
Loop of systems flow diagrams. 


4.4.1. Mason’s Gain Formula 


Mason's gain formula (1953), or Mason’s rule, is a method for deriving the transfer func- 
tion of a system through its SFG. By applying Mason’s rule there is no need to use block 
diagram reduction. The mathematical formulation is given in Equation 4.19: 


k 
T,D 


n n 


H(z) =-=1__ 


: (4.19) 


where T, is the gain of the n-th forward path 
D is the determinant of the diagram and is given by the following relation: 


D=1-YL4+ YL-Y 1st (4.20) 


with L, the gain of each loop, 

L, the product of any two non-touching loop gains, 

L, is the product of the gains of any three pairwise non-touching loops, and so on, 

D,, is the subdeterminant of the path T, and is calculated from Equation 4.20, without 
taking into account that part of the SFG that is non-touching with the n-th forward path. 

For example, let the difference equation of the discrete system: 


yln] = 2y[n-1] + x[n] + 5x[n-1]. 


In Figure 4.15, the SFG is illustrated. 
In Figure 4.16, the implementation structure of the system as a canonical form is 
illustrated. 


Source node 1 2 3 Sink node 
®@ ® & e @ 
X[n] 2 Y[n] 
or or 
X(z) 2 5 Y(z) 
7] 
‘ elay Branch 


FIGURE 4.15 
SFG of the difference equation. 
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x[n] y[n] 
FIGURE 4.16 
Canonical form implementation structure. 
Ss 
4.5 Solved Problems 
4.5.1 The following systems are connected in series: 
1 a 
H(z) = ——,H,(z)=1+2z_,H,(z) = ————_. 
Waar) 3(2) (+22) 


Determine the systems that need to be connected in parallel to lead to the same 
result as the connection in series. 


SOLUTION: H,(z), H,(z), H3(z) can be given in the following form: H,(z)= . H,(z)= 
1+z a= Zz . 
ge ey 


Connection of the above systems in series yields: 


5z-(z+1) 


ENTE) ENA ayy 


(1) 


Obviously, the systems to be connected in parallel will be computed by the 
technique of partial fraction decomposition of H(z). 


A(z) ___ 5(z +1) = A % B, ¢ B, = B, . 2) 
Z (z+3)\(z+2)?  (z+3) z+2 (z+2)° (z+2) 


The coefficients of the fractions are computed next: 


_ 5(-3+ 1) _ 5-2) _ 


_ Ai) 
7 aan) Cay 


Z 


_ 5(z+1) 


= 10 
(z+2)° 


A (z+3) 


z=-3 z=-3 
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ee a 5(z +1) 
wo 2dz | (z+3) Ls 


1d {5(z+3)-5(z+1) 1d{ 10 

2 dz (z+3) > 24% (E43 J, 
1 {-10-2-(z +3) 10 10 

2 = = =-10 
2 (z+3)* » et ay, «= (-2 +3) 


B, ~£{ HO es2y 


_d see) _ 10. 45 
wan dz\ (z+3) JL, (2 +3P - 


dz 
ie H(z) (+29) = (z+1)}  _ 5(-2 +1) _ 
Zz pap «= Z+3 |) )©6(-24+3) 
Thus, 
H(z) _ 10z ,(-10)z,_10z =a we (3) 
Zz Z+3 _Z+2, (z+2)° (z+2) 
Hi (z) H(z) H3(z) Hy (z 


From relation (3), it is concluded that the systems that need to be connected in 
parallel to lead to the same result as the connection in series are: 


1 1 
Hi(z) =10-—_——, H(z) = (-10): ——— 
i) [eae 2@)= C1) To 
H(z) =10-—= —_, Hi(z) = (-5)- 2 — 
: Gao 7 Gao? 


4.5.2 Let the block diagram of a discrete casual system be described as: 


x(n] y[n] 


192 Digital and Statistical Signal Processing 


Illustrate the zeros and poles of the transfer function H(z) and define the ROC. 
For which values of k is the system stable? 


SOLUTION: The implementation structure is of direct form II, so the transfer function 


i gs 

He milbe He = 
X(2) 14%, ack 

3 3 


k 
The ROC is: || > 3" The system is stable for |k|<3 


4.5.3 Let the transfer function H(z) of a discrete system: 


10(1- 2.3) (1- La 22") 
fiey= 3 2 


0-3") (2) [Pee] (Ed) "| 


Plot the block diagram for the system a) in cascade form and b) in parallel form. 


SOLUTION: The transfer function H(z) can be written: H(z) = 10H,(z)H,(z) 


(1-22+] [1-22] 
. 3 3 
where: H,(z) = 7 (1) 
(1-32"] (1-}e*] ee ce: 
4 8 8 32 
and 
(1-22 Ja-20 1eSzta2? 
H(z) = 


1 01 7 3 ~ a, lis 2) 
ea} [G-ahe] vet 


In the following figure the implementation of the system in a cascade form is 
illustrated. 


x[n] 
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For the implementation in parallel form, H(z) must be written in a form of a 
partial fractions sum: 


After some calculations, we get: 
k, = 2.93,k, =-17.68,k; = 12.25 —j 14.57,k, = 12.25 + j 14.57 


Thus, 


2.93, -17.68 | 12.25-j 14.57 , 12.25+j 14.57 
& 4 il, (3 2) : (3 4) ; 
to ——= | = eat 1 toss 1 
4° e lagi Plea? 


14.75 — 12.9z7 2 24.5 — 26.82 z7 
re Sete Sige j= zie Pee 
8 32 2 


H(z) = 


H(z) = (4) 


The following figure illustrates the implementation of the system in parallel form. 


-14.75 
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4.5.4 Let the discrete-time system of the following figure: 


x[n] z 


y[n] 


Zz Bea ee 


a. Design the realization structure of the depicted system using only one delay 
element. 

b. If the input is x[n] = u[n —- 2], find the output, y[n], of the system, using 
z-transform. 


SOLUTION: 
a. From the above figure, it is deduced: 


yln] = y[n—-1]+x[nJ+x[n- > 


yln)—yln—1] = a{n] + 22-11-27) = X@+24) > 


_ Y(z) _ 142" (1) 
“X(z) 1-23 


H(z) 


With the realization of the system as a direct form II, only one delay element 
is used as illustrated next: 


xin] yin} 


b. The z-transform of the input x[n] = u[n— 2] is: X(z)=z7U(z)=27 “a= Zz : (2) 
Za Za 
ore) zt+1 
But Y(z) = H(z)X(z)= = 
ut Y(z) = H(z)X(z) (z—-11-z7)_x(z—1 (3) 
sc ee (4) 


2 #¢-l) 2 2  g-1 @-i) 
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We compute the fraction coefficients: 


thus: 


3z Qz ET 
+ => 
z—-1 (z—-1) 


Y(z)=3+ : y[n] = 6[n]+ 6[n—1]- u[n]+ nu[n] > 


y[n] = 36([n] + 6[n— 1] + (2n- 3)u[n] (5) 
4.5.5 A discrete-time system is described by the input-output relation: 
yln] = x[n]+ y[n—-1]-0.1y[n- 2] 


a. Develop the system as a parallel connection of the systems H1(z) and H2(z). 
Then, give its schematic representation. 

b. Find and plot the impulse response of the system. Which pole determines the 
response of the system? 


SOLUTION: 
a. y[n]=x[n]+ y[n—-1]-0.1y[n—-2] (1) 


From the difference equation (1) of the discrete system, the transfer function 
is calculated in the form of equation (2), and the implementation diagram in a 
parallel form is designed. 


Y(z 1 z Zz 
() =H@=~9 = 7 ss = = 
X(z) 1-z"+0.1z z’>—z+0.1 (z—0.887)(z—0.113) 


1.146z 0.146z 


Be 2 
@)= 08872-0113 @) 
Hy (z) H(z) 
| H(z) 


+ 
X(z) Y(z) 
> 

oe 


—_—_————_> H,(z) aa 
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b. The impulse response of the system is calculated: 


h(n) = IZT{H(z)} =| 1.146(0.887)" — 0.146(0.113)" Jufn] (3) 
(3) = h[n] = h,[n]- h,[n] (4) 


The impulse response is plotted next: 


1.146 h1[n] 
y 
A 
h[n] | 
- 
3 4 5 


-0.146 


We notice that the term h,[n] has a much smaller contribution (in terms of values) to h[n] 
and is practically zero for n 2 4. Thus, the impulse response is approximated by: 


h{n] = h,[n] = 1.146(0.887)"u[n] , n>=4 


4.5.6 The discrete-time systems S1 and 82 are described in the following figures. 
a. Calculate the discrete impulse response of the first system. 


b. Compute the coefficients K1, K2, K3 and K4 of the second system, so as to have 
the same impulse response as the first. 
c. Give other forms of realization and compare each form with the others. 


d. Are the systems stable? 


System S1 
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System S2 


SOLUTION: 
a. For S1 (implemented in series), the total transfer function is: 
J-z? 1-2! 1-277 qorty¥ 
H(z) = H,(@)H2(2)H3(2) = — 1 _—= -! 1 (1) 
J-=z11-=z1 (1-227 )(1-27] 
(1) _ 493 
H(z) (z-1) 23,0, .2 . 
ebay PHT 
Z| Z 9 Z 3 p 5 
‘i 43 
Aste — G2) ne pag | GY | a 
| a 1) 230 dz ( at t) 
z—-—]|}z—-= tf aol 
2 8 2 8 
_1)/ 3 
C= (z-1) og, ay (z-1"_\_ 343 
Z= ( 1) zat at 3 
Z| 2-0 8| Z| Z-— 
8 
Thus, 
a, 343 
(2) ado 
H(z)=—18 112-3 Tid 3 ; @) 
g— goes 
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The discrete impulse response of the first system is given by relation (4). 


n(n] = IZT[ H(z) ]=-1128[n] 166([n— 1] (3) uin)+ (2) u[n] (4) 


b. For $2 (implemented in parallel), we can see that: 


¥(@) 
X(z) 


=k, +k,z' +k, ; +k, (5) 


Thus k, = -112,k, = -16,k, = -4/3, k, = -343/3 
c. Sl: Implementation in direct form II (Canonical form) 


1-3z'+3z*-z?  Y(z) 
1- ee + ae X(z) 
8 16 


H(z)= 


Based on the relation (6), the implementation of the discrete system in 
canonical form is presented in the following figure: 


x(n] y[n] 
a 
S1- Implementation in direct form I 
3 4, 1 os -1 2 2-3 
(6)> ae woes Y(z)=(1-3z7 +3z~ -—z~)X(z)> 


y[n] = x[n] — 3x[n—1]+32[n—-2]- x[n-3]+ 2 yln 1] * y[n— 2] (7) 
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Based on the relation (7), the implementation of the discrete system in 
direct form I is presented in the following figure: 


yin] 


y[n-2] 


A comparison Table 4.1 follows, which contains the number of memory ele- 
ments, adders and multipliers for the four different implementations. 


d. The poles of the systems are included in the unit circle, so the systems are 


stable. 

TABLE 4.1 

Comparison List 

Comparison In Parallel In Series Direct FormI Direct Form II 
Memory elements 3 3 5 3 
Adders 3 5 5 7 


Multipliers 6 5 5 5 
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4.5.7 The transfer function of a digital controller is given by 


G(z)= U(z)_ (1+ 0.25z) 
~ E(z) (1-0.5z)(1—-0.1z) 


Realize the above transfer function as a direct form, in series and in parallel. 
SOLUTION: 


=f7=1 2 -1 
G(z) = U(z) _ 5(1+ 0.2z) 5z (z~ +0.2) 5z° +2 


= — — = => 
E(z) (1-0.5z)(1-0.1z) (z*-0.5\(z"-0.1) z?-0.6z*+0.05 
20z +100z? 
G(z) = =] 2 
1-12z" +20z 


(1) = 0.05U(z) — 0.6z7U(z) + z°U(z) = 5z7E(z) + 27 E(z) > 
0.05u[n] = 5e[n — 2]+ e[n-1]+ 0.6u[n- 1]-u[n-2] > 

u[n] = 100e[n — 2] + 20e[n — 1] + 12u[n — 1] - 20u[n— 2] (2) 
Based on relation (2), the realization of the discrete system in direct form I 
is designed. For the design of the realization of the discrete system in cascade 
form, relation (3) is used, whereas relation (4) is used for the parallel form design. 


U(z) _ Zz Aas - 22° 


104+50z7 
E(z) 


= . 3 
z?-05 z't-01 1-2z1 1-10z7 co) 


U(z) _ zt+5 _ 17527 te 37.52" (4) 
E(z) (1-0.5z)1-0.2z) 1-2z? 1-10z7 


The realization in direct form I will be: 


e[n] 


e[n-1] 


e[n-2] 
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The realization in canonical form will be: 


e[n] u[n] 


e[n] ufn] 
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4.5.8 Derive the difference equation for the system described by the following SFG: 


Source Sink Node 


Node 


x[n] W.[n] y[n] 


SOLUTION: The SFG results in the following relations: 


w,[n] = x[n]+ aw,[n] + bw,[n] (1) 


w,[n] = cw,[n] (2) 


(1),(2) = w,[n] = cx[n] + acw,[n] + bew,[n] > 


wan] = 6) 
But y[n] = dx[n] + ew,[n] (4) 
Hence, the difference equation of the system is given by equation (5). 
(8) = ylal=(a+ Jot 6) 
1—ac — be 


4.5.9 Plot the SFG and derive the difference equation of the system described by the fol- 
lowing block diagram. 


a = | bo a 
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SOLUTION: The SFG is easily plotted using the block diagram of the system. 


bi 


Then, from the SFG, the following relations are derived: 


w,[n] = x[n] + aw,[n] 


w,[n] = bow,[n] + b,w,[n] 


w,[n]= w,[n— 1] 


y[n] = ws[n] 


(5) = y[n]= ‘ist Hosta +bw,[n—1] 


(2) = w,[n]= w,ln|=x{a]+ aw,[n—1] 


In z- plane it is: 


Y(z) = (by + byz™')W(z) 


W,(z) = X(z)+ azW,(z) = esce 
1l-az 
(8),(9) 2 Y@)= GbE) x¢~ 
1-az 
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(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


(10) 
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Thus, the difference equation is given by equation (11): 
y[n] = ay[n— 1] + byx[n] + b,x[n- 1] (11) 
4.5.10 Calculate the transfer function of the discrete-time system described by the fol- 


lowing SFG: 


-1 Win] & W2[n] 


X[n] y[n] 


Waln] Z ~ Woln] 


SOLUTION: First suggested solution: Derivation of the mathematical model describing 
the system. 


From the SFG we get: 
w,[n] = w,[n] - x{7] (1) 
w,[n] = ow,[n] (2) 
w,[n] = w,[n] + x{n] (3) 
w,[n] = w,[n - 1] (4) 
y[n] = w,[n] + w,[n] (5) 


Assuming zero initial conditions we z-transform the mathematical model: 


W, (Z) = W,(Z) - X(z) 6) 


W,(z) = aW,(z) (7) 
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W5(z) = W,(z) + X(z) (8) 
W, (2) = W,(z)z™ (9) 
Y(z) = W,(z) + W,(2) (10) 
(6) 
W,(z) = @W,(z) — a&X(z) (11) 
(8) 
W,(z) = ZW) (z) +z" X(z) (12) 


(12) 


W,(z) = a@z"'W,(z)+ az"X(z)—aX(z) = 


‘A (13) 
W,(z) = az X(z) =o) 
1l-az 
W,(z) = az 'W,(z)— a@z7'X(z)+z7'X(z) > 
z'X(z)-—az'X(z) (14) 


W,(2)= 


1-az" 


Therefore, from relation (14) the transfer function of the discrete system is 


derived: 
mb 
Then, the difference equation that describes the system is: 
h[n] = oe" 'u[n—1]- oc" u[n] (16) 
Second suggested solution: Using Mason’s gain formula 
y inp, 
Itis true that: H(z)= "1 = seta er EEE (17) 


D 
where: T,, is the gain of the n-th forward path that connects the input with the out- 
put. In the SFG of this problem, there are four paths. 


T, =-l-a, T,=z7, T;=-1-a-z7, T,=a-27 (18) 
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Dis the determinant of the diagram and is given by 
D=1- 1,4 VL,- YL, = 1-024 (19) 


D,, is the subdeterminant of T,, path and is calculated from (19), not taking into 
account the loops touching the n-th forward path. Obviously D,, = 1. 


Thus, the transfer function of the discrete system using Mason’s formula would 
be: 


“ —latz'+(-la-z")+a-z1  -a+z"! 


17)=> H(z) = = 2 
( ie @) 1-az" 1-az" 20) 
The result is the same with the one derived at the first suggested solution. 
er 
4.5.11 a. Implement the causal filter with transfer function H,(z) = oe with both 
IIR and FIR digital filter. 1-0.42 
b. Implement the causal filter with transfer function 
(=2")=2° =1.627) 
H(z) = H,(z)- H(z) = 
ie a (1-0.4z*)(1-0.8z7) 
SOLUTION: 
— rl 
a. Realization with IIR digital filter: H,(z) = bs ae (1) 


Xz) 1-0Aaz" 

The above filter is stable as all of its poles (a=0.4) are included in the unit circle. 
Thus, from equation (1), the difference equation is obtained as follows. We first 
convert it to: Y1(z)-(1 - 0.4z-3) = X1(z)-(1-z-1) >Y1(z) - 0.4Y1(z)z-3 = X1(z) — X1(z)z-1 

Since the terms of the form Y(z)z7 correspond to the inverse z-transform of the 
form y[n — 1], we can write: 


y,[n]- 0.4y,[n — 3] = x,[n]— x,[n—- 1] = y,[n] = 0.4y,[1— 3] + x, [n] - x, [0-1] (2) 
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From equation (2), the filter desired can be implemented in canonical form: 
Direct form H-Canonical form 


Y[n-3] 


Realization with FIR digital filter: i.e., realization with a FIR filter of the form: 


We only consider the ten most important terms. We write the transfer function 
of the filter to the following form: 


1-2" _ 2a—z 


A=) pase Pood @) 


By dividing (using only the first ten most important terms) the polynomial of 
the numerator and the polynomial of the denominator, the following is derived: 


H,(z)=1-z71 +0.4z% —0.4z* + 0.16z* — 0.16z” +0.064z2° = 


Yi( 


as 1—z140.42° - 0.424 +0.16z° —0.16z7 +0.064z° = 


H,(z)= X,(2) 


Y,(z) = X,(z)—2z'X,(z)+ 0.42 °X, (z) — 0.42 7X, (z) + 0.162 °X,(z) 
—0.16z’X,(z) + 0.064z°X,(z) 


Since the terms of the form Y(z)z~ correspond to an inverse z-transform of the 
form y[n-i], we obtain the corresponding difference equation: 


y,[n] = x,[n] — x,[n- 1]+ 0.4x,[n- 3] — 0.4x,[n — 4]+ 0.16x,[n — 6] 


—0.16x,[n—7]+ 0.064x,[n-9] 
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The realization of the filter is illustrated in the following figure. By comparing 
the two different ways of implementation of the filter (IIR and FIR), one can easily 
notice that the IIR filter is much simpler (and consequently cheaper) than the FIR 
filter. 


X[n] 


Y[n] 
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(l-z7?)(l-z?-1.6z7)  1-2z'-0.6z74+1.6z° 
(1-0.4z2°)(1-0.8z")  1-0.8z7-0.4z2°+0.32z* 


b. H,(z)=Hy,(z)-H2(z)= 


Since, H(z) = am => 
3 


Y3(z)(1— 0.827 — 0.42% + 0.32z~) = X3(z)(1— 2z7 — 0.6z* + 1.6z°) > 


(and after applying inverse z-transform) 


y3[1] = x3[n] — 2-x3[n—1]-0.6x3[n— 2]+1.6x3[n-3]+0.8-y3[n— 1] 
+0.4y,[1— 3]-0.32y,[n — 4] 


The realization is illustrated in the next figure: 


Y[n-3] 


x(n) ! Y{n) 
I 
Yy v 
Ze \ ee z 
ve 
| 
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zi i z 
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Frequency Domain Analysis 


5.1 Introduction 


Fourier analysis is an important tool for extracting significant information of the frequency 
domain of a signal that is not apparent in its representation in the time domain. It is used 
for signal analysis, describing the spectral behavior of the signal. Since the spectral infor- 
mation is given within the frequency domain, the Fourier transform domain is called the 
frequency domain. 

From the signal and the noise spectrum, it is possible to draw useful conclusions, 
such as: 


1. The frequency range which the sinusoidal waveform components, composing the 
signal on the whole or most of it, are located in. 


2. The frequency range which the sinusoidal waveform components, composing the 
noise of the signal are located in. 


3. The type and source of noise. 


4. The frequency range that can be cut off to reduce the presence of noise to the mini- 
mum possible, without significant distortion of the signal itself. 

5. The minimum possible sampling frequency that does not introduce artifacts, 
according to the Nyquist theorem, in cases when sampling is implemented. 


Each discrete-time signal can be represented as a sum of properly shifted impulse func- 
tions. In this section, an alternative representation of discrete-time signals will be studied 
as a combination of complex exponential sequences of the form e7*". 

The following are going to be analyzed: 


e The Discrete-Time Fourier Transform (DTFT), which is applied to non-periodic 
discrete-time signals. 


e The Discrete Fourier Series (DFS), which is applied to periodic sequences. 


e The Discrete Fourier Transform (DFT), which is applied to non-periodic discrete- 
time signals of finite length. 
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e The Fast Fourier Transform (FFT), which is an algorithm for calculating the DFT. 
e The Discrete Cosine Transform (DCT). 
e The Discrete Wavelets Transform (DWT). 


5.2 Discrete-Time Fourier Transform (DTFT) 


The discrete-time Fourier transform of a discrete-time signal, x(n), is the representation of 
this signal as a combination of complex exponential sequences of the form e7", where @ is 
the angular frequency in rad/sec. The original signal can be computed when its DTFT is 
given, by means of the inverse discrete-time Fourier transform. 

The DTFT of a sequence x[n] is defined as: 


X(e!) = Fixtal} = Y° xfn|e™ 6.1) 
whereas the inverse DTFT is defined as 


=—° tx( ele)t = 3, J« el) ei" day (5.2) 


Equations 5.1 and 5.2 form the pair of discrete-time Fourier transforms. Equation 5.1 is 
called an analysis equation and expresses the analysis of the discrete-time signal, x[n], in 
exponential signals e7” that extend into a continuous range of angular frequencies w lim- 
ited in the interval 0 < w < 2 a, while Equation 5.2 is called a synthesis equation. The suf- 
ficient condition for the existence of DTFT is for x[n] is an absolutely summable sequence, 


ie., y Ix{n]| <0, 


n=—co 


Table 5.1 includes the DTFT of basic signals, while Table 5.2 includes some of the proper- 
ties of the DTFT, which are used to analyze discrete-time signals and systems. 

Parseval’s theorem refers to the conservation of energy during the transition from the 
time domain to the frequency domain. The quantity xe) is called energy-density 
spectrum of the discrete signal, x[n], and describes the distribution of the energy of the 
discrete signal, x[n], over a frequency range. 
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TABLE 5.1 
DTFT of Fundamental Signals 
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Signal 

6[n] 

d[n — N] 

u[n] 

a@uln], |a| <1 
sin(@ i) 
sin(@on + 8) 


cos(@i1) 


cos(@on + 0) 


u[n] — u[n - M] 


1 n=0,1,2,... 
1 n=-1,-2,... 


2a 


sgn[n] = | 


DTFT 
1,-0.0 <aw@<o 


eIN N=+41,+2,... 


1 
1-e” 


+ > 6(@ — 27k) 


ke-00 


a re) + 2ak)- 5 ) + 20k 
5 Le @) +22k)-d(@+@ mk) | 


k=-00 


L jx [ “5 + @y — 2ak) - 51 -@)- ank) | 
Reco 


ry [oo @y + 27k) + 5(@ + @y + 2k) | 


k=-c0 


ya [ te + @, — 2k) + PLCS -a,- ont) | 
— 


Sin@M2) sor 
sin(@/2) 


1-e 


y P2p(@ + 2rk) 


kee 


sin [(4 +2) o| 


sin(«/2) 


Where Pil] = Poy alnl 


(Rect. Pulse) 
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TABLE 5.2 
Properties of DTFT 
Property Non Periodic Signal DTFT 
x,[n] x,(e a ) i aad é : . 
periodic functions with period 
x,[n] x,(e) of 2x 
Linearity ax[n] + by[n] aX(e!) + bY(e) 


Time shifting 
Frequency shifting 
Conjugate sequence 
Time reversal 

Time expansion 


x[n -— No] 
ely] 
x*[n] 
x[-n] 
x(n/k) for n multiple of k 


en fond X (ei) 
X (ei? — ©) 
X'(es") 
X(e) 
X (ei) 


Convolution x,[n] * x,[n] X,(e”) X,(e”) 
1 ; 
Multiplication x[n]x,[n] on J X,(e*)X, (el) d0 
2n 
jo 
Derivative in frequency nx[n] j me ) 
X(el?) =X (e”) 
Re{x(e”)} = Re{ x(e)} 
Conjugate symmetry for real signals x[n] real Im{X(e” y} = Im{X(ei)} 


x(e”)| =|x(e”) 
ZX(el) =-ZX(e!”) 


x[n] real and even 
x[n] real and odd 


Symmetry for real and even signals X(e) real and even 


Symmetry for real and odd signals X(e) imaginary and odd 


Dil x09 P= 5 f1X(e*) Fao 


Parseval’s Theorem 


n=—00 


5.3 Discrete Fourier Series (DFS) 


Let the periodic sequence x[”] with period N. Obviously it is true that x{n]=x[n+rN] 
(where n, r are integers). As in the case of continuous-time signals, a periodic sequence can 
be represented by a Fourier series, that is, by a sum of complex exponential terms whose 
frequency is an integer multiple of the fundamental frequency, 22/N, as shown in the fol- 
lowing relations: 


k 
Synthesis Equation : x[n] = X[kle (5.4) 


Unlike the case of continuous time signals, the Fourier series representation does 
not require an infinite number of harmonics (multiples of the fundamental frequency), 
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but only N complex exponential terms. The reason is that the complex exponential terms 
are periodic to N: 


jon N, jam kn jam jon kn jontn 
Ceawin}=e N =e Ne N =e NeP™mie’ N =e {n} 
where | is an integer. 
The coefficients of the Discrete Fourier series are given by: 
N-1 _ ok 
: - ~ a j2n—n 

Analysis Equation: X[k] = x[nje (5.5) 

n=0 


The pair of Equations 5.4 and 5.5 define the Discrete Time Fourier Series of the discrete- 
time periodic signal, x[n]. The coefficients X[k] are called Fourier coefficients or spectral 
lines. 

By designating 


Wee" (i) 6.6) 


N-1 
>’ XM Wo"* — 1=0,1,...,.N-1 (67) 


XIk]= Yin] Wik =k =0,1)...,N-1 (5.8) 


5.3.1 Periodic Convolution 


Let the periodic sequences %,[n] and x,[n] with a period, N, and with discrete Fourier 
series components X,[k] and X,[k], respectively. 
The sequence: 


Hn] = S&L mz, [n— ml] (59) 


is the periodic convolution of x,[n] and x,[n], and the components of its discrete Fourier 
series are given by the following relation: 


X[K] = X,[k]X,[k] (5.10) 
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5.3.2 The Relation of the DFS Components and the DTFT over a Period 


Consider the periodic sequence, x[n], with the components of the Discrete Fourier Series 
represented as X[k] and the Discrete Time Fourier Transform over a period of the sequence 
x[n], represented as X ei). In this case, the following relation is true: 


Rk]=X(e)| (5.11) 


o=2n— 
N 


5.4 Discrete Fourier Transform 


The Discrete Fourier Transform (DFT) is used for the frequency representation of finite- 
time signals. In fact, what is sought is the calculation of the DTFT, but what can actually 
be calculated is the Discrete Fourier Transform (since DTFT is a continuous function of 
the discrete frequency ). Since the DFT is obtained by sampling in the DIFT frequency 
domain for finite-length signals, the sampling effects must be made completely clear. The 
Discrete Fourier Transform, apart from its importance for the representation of discrete- 
time sequences in the frequency domain, is the basis of many applications of signal pro- 
cessing, such as spectrum analysis and data encoding (JPEG, MP3, MPEG video). 

Additionally, in many cases (such as in spectrum analysis and filtering applications), the 
signals are not of finite lengths, and the DFT implementation requires the use of a time 
window, the processing of frames, or the use of an alternative DFT form suitable for time 
variable signals, which is called Time Dependent Discrete Fourier Transform. 

The Discrete Fourier Transform (DFT) is an alternative form of the Fourier transform in 
which the finite length sequences are transformed into other sequences, which represent 
samples in the frequency domain and are defined by the relations: 


N-1 ik N-1 
X[k] =} xInle Pe Sing (5.12) 
n=0 n=0 
1 N-1 jon 1 N-1 
~1N yne?*n" = 2 w- 5.13 
xnl= 5; 2 [ke vd i (6.13) 


27 
where Wy = e N.The factors W," are called twiddle factors and are the complex sequences 
on which the DFT is based. Note that the following is true: 


WY =e N™ =e" =1=We (6.14) 
Generally, it is true: 


Wo =n, (5.15) 
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X(e"”) 


\ oe a~ 


mr 0 kAw nr Pl Aw l€ an w 


FIGURE 5.1 
Sampling at the DIFT frequency. 


The procedure of sampling at the DTFT frequency is illustrated in Figure 5.1. 
The DFT of Equation 5.12 in the form of a system of algebraic equations is: 


X[0] = x[O]Wy + x[1]Wy + x[2]Wy +... 
X[1] = x[OJWy + x[1]Wy + x[2]We +... (5.16) 
X[2] = x[O]WA + x[1]Wy + x[2]Wy +... 


For example, in the case of N = 3 samples the DFT matrix is: 


X[0]] |W; W; Ws |[ x{0] 
X[1]} |=|W; W; We | xf] (5.17) 
X[2]}| |W? wy ws | 212] 


where W,' = W,'*! = W;.. Substituting in Equation 5.17, the following is obtained: 


xfoy] [1 1 1 x(0] 
X[1] |=]1 0.5- 0.867 — 0.5 + j0.867 | | x{1] (5.18) 
X[2]| |1 -0.5+ 0.867 0.5— 0.867 | | x{2] 


A method of calculating the N-point DFT is based on factorizing Equation 5.12 to reduce 
complex multiplications. 


X[k] = x[0]+ x[1]We + x[2]wi* + x[3]We* +..+X[N — qwen-) 
xfO]+ WA x{1] + x12] + x[3]W2* +... +x1N — WES | 


(5.19) 
= X[k] = x[0]+ wi Ea + WA | xf2]+ Wé [x13] +..WéxtN - 11] 
The sequences x[n] and X[k] are periodic with a period N, that is, 
x[n+ N]=x[n]Vn (5.20) 


X[k + N]= X[k]Vk (5.21) 
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5.4.1 Properties of the DFT 


Some of the properties of the Discrete Fourier Transform are proportional to the corre- 
sponding properties of the Discrete-Time Fourier Transform, but there are other different 
properties, as well, due to the finite length of the sequences and their Discrete Transform. 


5.4.1.1 Linearity 


DFT| ox, [n] + bx[n] |= DFT x,[n] ]+ bDFT [ x,[n] |= oX,[k] + bX,[k] (5.22) 


5.4.1.2 Circular Shift 


To understand the circular shift, consider the finite sequence x[n] (n = 0,1, ..., N- 1), which 
is set on the circumference of a circle clockwise and at equal intervals. Then, x[n — k] is the 
clockwise rotation of the circle for k interval. 

A circular shift of the sequence x[n] = {x[0], x[1], x[2], ..., x[N — 1]} to the right is math- 
ematically stated as x([n — 1] mod N), and it denotes: 


x([n-1]mod N) ={x{N ,x{1],...,x[N —2]} (5.23) 
Two circular shifts to the right, i.e., x[n — 2], create the sequence: 
x([n—2]mod N) ={x{N-2],x[N -1],x{0],x{1]...,x[N -3]} (5.24) 
N right shifts recover the original sequence,x[n], thus, 


x([n— N]mod N) ]={x[0] x[N -1]} (5.25) 


DFT of the circular shift in time domain: 
It is true that: 


DFT| x[n-m] mod N |= Wx" X[k] (5.26) 


Equation 5.26 indicates that the DFT of a circularly shifted sequence is of the 
same magnitude but of a different phase with the corresponding DFT of the origi- 
nal sequence. 


DFT of circular shift in frequency domain: 
It is true that: 


DET [Wy?"x{n] |= X[k - a]mod N (5.27) 


Equation 5.27 indicates that the multiplication of the sequence x[n] with the 
exponential sequence Wy" is equivalent to a circular shift by a units in the fre- 
quency domain. 
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5.4.1.3 Circular Convolution 


Consider two sequences x[n] and h[n] of length N. x[n] ® h[n] denotes their circular convolu- 
tion and it is defined as follows: 


x[n] ® h[n] = S x((n —k]modN)h{k]= y x[k]h([n- k] mod N) (5.28) 


k=0 k=0 


When the two sequences are of different lengths, N, and N,, respectively, then the smaller 
of the two is filled in with zeros so that both get to the same length, N, where N = max(N,, N,). 
The circular convolution is of length M = max (N1, N2). 

The steps for the calculation of the circular convolution of two sequences are: 


1. Circular reflection of one sequence 
2. Circular shift of the reflected sequence 


3. Element-by-element multiplication of the reflected and shifted sequence with the 
other sequence and summation of the products. 


DFT of the circular convolution 
The DFT of a circular convolution of two sequences of finite length is given by: 


DET x{n] ® h[n] |= DFT x{n] |DFT| h{n] |= X[KH[K] = Y[K] (5.29) 
Thus, the response of the discrete system will be: 


yln]= IDFT [ X[k] H[k]] (5.30) 


Equation 5.30 shows that the DFT can be used to calculate the linear convolution by 
increasing the length of the sequences, x[n] and h[n], to N (by adding zero elements to 
the end of each one) and performing circular convolution of the resulting sequences. The 
result is the same as the one obtained by the linear convolution of the original sequences. 
The illustration of the procedure of linear convolution with the use of DFT is shown in 
Figure 5.2. 


x(n] X(k) 
—>| DFT 


X(KQH(K) ar Velexindete) 


hin] H(k) 
—>|__DFT 


FIGURE 5.2 
Linear convolution using DFT. 
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5.4.1.4 Multiplication of Sequences 


DFT x,{u}-x3[n]]=—X,Ik] ® XK) (531) 


Based on Equation 5.31, multiplication of two sequences in the time domain corresponds 
to the circular convolution of their DFTs. 


5.4.1.5 Parseval’s Theorem 


It is true that: 


N-1 5 1 N-1 5 
Daren = 5 DO) 6.32) 


n=0 


Parseval’s theorem expresses the conservation of energy in the transition from the time 
domain to the frequency domain. That is, the sum of the squared values of the samples is 
equal to the average value of the squares of the spectral lines. The |X(k)|* representation is 
called power spectrum of x[n] and depends only on the spectrum magnitude and not on 
its phase. 

In Table 5.3, all properties of the Discrete Fourier Transform are listed. 


TABLE 5.3 

DFT Properties 

Property Sequence of Length N N Point DFT 
x,[n] X[K] 
x.[n] X[k] 

Linearity a,x,[n] + a,x,[n] a,X,[k] + a,X,[k] 

Circular shift in time x[<n -— Np >] WwW X[k] 

Circular shift in frequency W ~'x[ 11] X[<k — ky >y] 

Conjugate sequence x*[n] X[<-k > x] 

Reflection in time x[<-n > y]J X[k] 

Circular Convolution x,[n] ® x,[n] X[k] X[k] 


Multiplication 


Conjugate symmetry for 
real signals 


Parseval’s Theorem 


x,[n] x,[n] 


x[n] real 


Sta PD ; 


1 N-1 
yD XilXals k—-m>y] 
X[k] =X *[<-k >y] 
Re{X[k]} = Re{X[<-k > ]} 
Im{X[k]} = —Im{X[< -k > ]} 
| X[k] =| XI<-k > y]| 
ZX[k] = -ZX[< -k >y] 
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5.5 Fast Fourier Transform 


The Fast Fourier Transform (FFT) is an efficient algorithm for calculating DFT. In practice, 
there is not only one algorithm but a variety of algorithms that can achieve the same result. 
The differences among them are mainly in the number and type of operations, as well as 
in the size of the required computer memory. 

When the sequence x[1] is known, then the DFT X[k] can be derived by using Equation 
5.12 or the inverse by using Equation 5.13. In order to calculate the sum for a k value in 
Equation 5.12, N complex multiplications and (N-1) complex additions are required. For N 
values of k, M complex operations are required where: M = N[N + (N - 1)] = N? (that is, N? 
multiplications and N - (N — 1) additions). The same is true for the IDFT calculation. For 
large N, there is a huge amount of complex operations, which creates many difficulties in 
the calculation of the DFT or IDFT of the x[n] and X[k] sequences, respectively. 

These difficulties are diminished with the implementation of the Fast Fourier Transform 
and the Inverse Fast Fourier Transform (FFT and IFFT, respectively). The method to be 
presented next was introduced by Couley and Tukey (1965) and is based on the technique 
of breaking up the original DFT into smaller segments, which, in the end, are reunited to 
create the DFT. 


The basic FFT method reduces the number of multiplications from N? to logon = ae 
where N = 2°. For example, if N = 1024 = 2”, then for the calculation of the DFT N? = 2” = 1.048.576 
complex multiplications are required, while the FFT requires only a Ue 5120 


multiplications. When N is a power of 2, then the algorithm is called radix - 2 FFT. 


5.5.1 FFT Equations 


Let the sequence x[n] have a length of N. It is divided into two segments of a length each, 
containing even-indexed and odd-indexed samples, respectively, i.e., 2 


x[n] = x,[n]+ x, [1] (5.33) 
where: 
x,[n] = {x{0], x{2], x{4],...,x[N - 2]} 
or (5.34) 
x,[n]=x[2n] , n=0,1,2, a 
and 


x,[1] = {x{1], x[3], x15], ..., x[N - 1} 


x[n]=2x{2n+1] , n=0,1,2...., ~ a4 (5.35) 
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N-1 
The DFT of x[n] is: DFT[ x(n) ] =} xtnlv", k=0,1,2,...,N-1, therefore, due to 
n=0 


Equations 5.33, 5.34 and 5.35, it is: 


Ny N, 
2 2 

X[k]= S) x12n]w" nl Yi x20 + wenn (5.36) 
n=0 n=0 


20 
-j=2 pa 
But, Wi=e “ =e N and wie 2 =e’N thus, 


Wy = Ww (5.37) 


X[k]= Six Lae $ Wa >) xolnlWshk =, 2p = (5.38) 


n=0 2 n=0 2 


Equation 5.38 shows that the DFT of x[n] is equal to the sum of the DFT of length N/2 of 
the even-indexed elements of x[n] and the DFT of length N/2 of the odd-indexed elements 
of x[n]. That is: 


X[k] = X,[k]+WEX.[k] , k=0, Lye (5.39) 


where X,[k] = DFT[x,[n]] and X,[k] = DFT[x,[n]] of length N/2. 
X[k] and X,|k] are periodic functions with a period > that is: 


(5.40) 


Ny aes | 


N i 2 (ky = Aik ne 
For example, X, lk + Y | = 2 Xk = Sy) XL! Ww. 


2: n=0 2 2 
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N 
_j2an N My 
J 2 


But W,2=e 2 =e?"=1=>X, [e+ N] = YS) XL = X,[k] 
Pa n=0 2 


2 


N 

Hence, x|k+ N] =X [k]+Wy 2 X,[k]k=0,1, poeraen GA) 

iN _ 27 |, N _j2tk _ 2k 

But since Wy 2 ai inl ee "N eit = ¢ 'N 
aN _ on, (5.42) 

=W, 2 "N” = Wh 
By combining Equations 5.41 and 5.42, the following relation is derived: 

x| +N ]=x,0k1-WeX) k=012,..5--1 (5.43) 


Equations 5.39 and 5.43 constitute the fundamental equations of the radix-2 FFT algorithm. 
The number of complex multiplications required for the DFT of x[n] (by implementing 


Ny .(™)-= N(N +1) 
2 2 2 
The process continues by breaking up the sequences x,[n] and x,[n] of length o in 


2 
only one stage of analysis) is: ( + : 
sequences with odd-indexed and even-indexed samples as follows: x,,[n], X,.[1], X,[n] and 
XM] with n=0,1,.. “y —1. Inthis second stage of segmentation, and based on Equations 


5.41 and 5.42, the following relations are derived: 


X[k] = X,.[k]+WakX,, [kK] , k=0,1, pg 1 
< (5.44) 
X[k] = X,.[k]+ Wek X[k] , k=0, U2 1 


and, respectively 


x, kN |= xf -We XA , B= OL 2p 
(5.45) 


x,[e+N |= xt Wax, : k= 01,2001 
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(00,011, 2,03, 014, Os, OL6, Ql7) 


FIGURE 5.3 
Iterative FFT algorithm. 
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FIGURE 5.4 
Two-point butterfly. 


In this way, the segmentation process continues until the sequence reaches to the length 


of one sample (Figure 5.3). Since N = 2”, the partition stages are v,and v stages require v- a 
multiplications. 2 

This process can be illustrated by means of the so-called ‘butterfly’ graph of Figure 5.4. 
It is the depiction of the calculation of a two-point DFT. 

For the case where the number of samples is N=8, the depiction of the DFT is given in 
Figure 5.5. The calculation of the 8-point DFT has changed to the calculation of two DFTs 
of N/2 = 4 elements each and the final combination of their results. The analysis proce- 
dure previously followed may continue for both new DFTs of N/2 points. So, we continue 
the analysis into even- and odd-indexed elements, and following the same technique we 
obtain the diagram of Figure 5.6. The process continues until it results in a 2-point DFT 
calculation (Figure 5.7). This will be done after m stages, where m = log2N At each stage, 
there are N/2 butterflies, that is, (N/2) log2N butterflies on the whole. 

The flow diagram of an 8-point FFT is shown in Figure 5.8, which suggests that one 
complex multiplication and two complex additions are required to calculate each butterfly. 
Thus, the N-point FFT requires (N/2) log2N complex multiplications and N log2N com- 
plex additions. The arrangement of the output samples is ‘proper, i.e., X[0], X[1], ..., X[7], 
whereas the arrangement of the input samples, x[0], x[4], x[2], x[6], x[1], x[5], x[3], x[7], is 
not in the right turn. This kind of arrangement of the input samples is the result of their 
gradual decomposition applied during the development of the algorithm. This decomposi- 
tion process is called decimation in time (DIT). The arrangement of the input elements is not 
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o<—— 3" Stage ———po 


x(0] X[0] 
x[2] x1] 
x[4] X[2] 
x[6] X(3] 
x(1] xi4] 
sa X(5] 
~) X{6] 
x7] > a X(7] 


FIGURE 5.5 
Third stage of development of the 8-point FFT algorithm. 


O<—— _ 2" Stage ———_ po 


x(0] X1[0] 
x[2] X1[1] 
x[4] X1[2] 
x[6] X1[3] 
x[1] X2[0] 
X2[1] 


x[5] 


x[3] X2[2] 
N/4 point 
x[7] oe X23] 
FIGURE 5.6 


Second stage of development of the 8-point FFT algorithm. 
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O<——_ 1 Stage ———>0 


x(0) X11(0) 
x(4) X11(1) 
0 1 
x(2) W, Xi2(0) 
x(6) X12(1) 
0 1 
x(1) W, X21(0) 
x(5) X21(1) 
1 
x(3) X22(0) 
x(7) X22(1) 
“1 
FIGURE 5.7 
First stage of development of the 8-point FFT algorithm. 
o<q— 1% Stage ——po o<q— 2" Stage ——po o¢— 3” Stage —po 
° LS ° 
x(4) > X(1) 
if) : 
X(2) X(2) 
x(6) > > X(3) 
we 4 
X(1) e xX(4) 
X(5) —> > 
wa 4 x(5) 
X(3) x(6) 


FIGURE 5.8 
Final flow graph of 8-point FFT. 


X(7) 


random, but results from the ‘proper’ arrangement of samples with bit reversal. The full 
match of ‘proper’ and reversed indices for N = 8 is given in Table 5.4. 

The FFT algorithm is a very fast and efficient algorithm, and when applied, the gain 
in processing time is undeniable, as demonstrated for various numbers of samples in 


Table 5.5. 
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TABLE 5.4 
Match of ‘Proper’ and Reversed Indices for N = 8 


Arrangement Resulting from 


‘Proper’ Arrangement Bit Reversal 

Decimal Binary Binary Decimal 

0 000 000 0 

1 001 100 4 

2 010 010 2 

3 011 110 6 

4 100 001 1 

5 101 101 5 

6 110 011 3 

7 111 111 7 
TABLE 5.5 
Complex Multiplications for DFT and FFT Calculations 

DFT FFT Ratio of Complex 
Number of Complex Number of Complex Multiplications 

N Multiplications Multiplications (DFT/FFT) 
2 + 1 4 
4 16 4 4 
8 64 12 53 
16 256 32 8,0 
32 1024 80 12,8 
64 4096 192 21,3 
128 16 348 448 36,6 
256 65 536 1024 64,0 
512 262 144 2 304 113,8 
1024 1 048 576 5 120 204,8 
2048 4194 304 11 264 3724 
4096 16 777 216 24 576 682,7 
8192 67 108 864 53 248 1 260,3 


The FFT algorithm, which has just been developed, is applied only to a number of 
samples that are power of 2 (radix-2) and not for a random number of samples, as 
for the DFT. There are algorithms that are applied only to a number of samples that 
are power of 4 (radix-4), or algorithms only for real-value inputs (real-valued algo- 
rithms), or algorithms for numbers of samples that are prime numbers (prime length 
algorithms). 

The previous analysis was based on the decimation in time (DIT). In a similar way, there 
is the decimation in frequency (DIF) technique resulting in an FFT algorithm (Gentleman 
& Sande, 1966), which has the same computational complexity as the corresponding DIT 
algorithm. A characteristic of the DIF algorithm is that the multiplications in the ‘but- 
terflies’ are made after adding the input values and that the input samples are in ‘proper’ 
arrangement, while the output samples are bit-reversed. 
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5.5.2 Computation of the IDFT Using FFT 


N-1 


For the computation of IDFT, which is obtained by x{n]= IDFT | X[k] |= ~ > XW", 
k=0 


20 2mk 
where Wy = e ’N, the FFT algorithm can be applied. At first, Wi = eX and the output 


sequence _ are defined where k = 0, 1, ..., N— 1. Then, the output of the FFT algorithm 


will be the sequence, x[n]. 


5.5.3 Fast Convolution 


The result of the convolution of two sequences h[n] and x[n] is calculated by: 
y[n] = x[1] ® h[n] = IFFT | FFT[x{n]]- FFT[h[n]] |= IFFT| Xk] x HIk] | 


Let N, represent the length of the sequence x[n], and N, represent the length of the 
sequence h[n]. After the linear convolution of h[n] and x[n], the length of y[n] is N, + Nj -1. 
The addition of zeros at the end of the sequence increases the length of h[n] and x[n] to M, 
where M > (N, + N,- 1) and M = 2” in order to make use of radix-2FFT. Obviously, for the 
calculation of y[n] the M-Point FFT is required three times (two times for the calculation of 
X[k] and H[k] and once for the calculation IFFT[X[k] x H[K]]). 

In the case that one of the sequences, /i[n] or x[n], is much longer than the other, the algo- 
rithm becomes slower due to the increment of the shorter sequence with zero samples. To 
avoid this problem, the following methods are applied: 


a. Overlap and add 


b. Overlap and save 


5.5.3.1 Overlap and Add Method 


Let sequence hi[n] be of length N, and sequence x[n] be of a much longer one (which could 
be practically infinite). x[n] is divided into segments of length, M - N + 1 (where M > N 
and M = 2°), and, then, the N — 1 zero samples are added at the end of each subsequence so 
that its length becomes equal to M. In Table 5.6, the values of Musually chosen are given 
as a function of N. 


TABLE 5.6 

Values of M Given as a Function of N 
N M=2" 

<10 32 

11-19 64 

20-29 128 

30-49 256 

50-59 512 


100-199 1024 
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The length of the sequence h[n] is increased to M by adding M — N zeros at the end. The 
circular convolution of h[n] with the segment of x[n] is calculated by an M-point FFT, and 
its length is M. It is: M = (M-N + 1) + (N-1), where M- N + 1 the first samples of the cir- 
cular convolution, and N — 1, the last samples of the circular convolution to be added to the 
next ( N — 1) samples in order to create the linear convolution. 


5.5.3.2 Overlap and Save Method 


In this method, the length of the sequence h[n] is increased to M by adding M — N zero 
samples at the end. x[n] is divided in segments in such a way that the length of each seg- 
ment is equal to M (M < N and M = 2”), and its first N — 1 samples are coming from the 
immediately preceding segment, i.e., they are overlapping it by N - 1 the samples. 

The linear convolution, x[n] * h[n], is derived by the successive circular convolutions of 
the segments of x[n]with h[n], ignoring, however, the first N — 1 elements, which results 
in the formation of a sequence consisting of their M- N + 1 remaining samples. 


5.6 Estimation of Fourier Transform through FFT 


Let the Fourier Transform (FT) of a continuous-time signal x(f), that is, X(). The process 
is as follows: 


a. We sample the signal x(f) with a sampling period T, resulting in the discrete-time 
signal x[nT],n =0,1,...,N-1. 

b. We calculate the DFT X; (using FFT) of the discrete-time signal, x[nT]. 

c. We can calculate the FT X@) at frequencies Q, = omy =0,1,...,N—1, according 
to the relation: NT 


20k 
j= 


ox N 
X(Q,)= nrice 


“oni X,,k=0,1,.....N-1 (5.46) 


The larger the N (i.e. the number of samples we get), and the lower the sampling fre- 
quency, the better we can approach the MF X(Q) of the continuous-time signal x(f) from the 
sequence X(Q,) of Equation 5.46. 


5.7 Discrete Cosine Transform 


The discrete cosine transform is a method that is effectively implemented in digital image 
compression. With the DCT transform, the information contained in an image can be 
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transferred from the time domain to the frequency domain (which is an abstract domain), 
where its description occupies significantly less memory space. The first frequencies in the 
set are of the greatest importance, and the last of less importance. Decisions on compres- 
sion of a part of these last frequencies (resulting in the loss of them), affect directly the 
tolerance standards set for image quality. The algorithm is applied in various MPEG and 
JPEG encodings. 

The DCT transform is defined as follows: For each pixel (x, y) 


DCT(i, j)= per cy Prete y)cos] CX*DI Joos] C+ Die (6.47) 


x=0 y=0 


where C(x) = 0.7071, x = 0 and C(x) =1,x>0 

Procedure: The DCT value (i, j) is obtained, that is the value of the transform coefficients 
in the frequency domain. Hence, pixel values correspond to coefficient values. Each of 
these coefficients contains a piece of the original information (corresponding to the part 
of the spectrum it describes). It is known that the human eye perceives phenomena asso- 
ciated with low frequencies (such as specific colors) much better than other involving 
high-frequency areas of the signal (e.g., edges of the image). This is why the transform 
coefficients corresponding to low frequencies have greater weighting factors than those 
describing the high frequencies, and why the first ones are described with maximum 
accuracy. During reproduction the inverse process is applied using the Inverse Discrete 
Cosine Transform (IDCT), which is given by the following equation: 


pixel(x, y) = = SY caper jycos| ODF Jens] C¥+ Die | (5.48) 
i=0 j=0 


The result is that the original information is taken back almost intact (except some 
unavoidable rounding errors). 

Thus, in the DCT, the information that the image contains can be transferred from the 
time domain to the frequency domain where its description can be achieved with a signifi- 
cantly smaller number of bits. After applying the transform to the image, the information 
of the result consists of numbers indicating the weighting factors of each frequency at the 
reconstruction procedure of the image. 

The DTC can be computed using the DFT, as follows: Let a sequence, x[n], of N 
length, ie.,0 <n <N-1. At first, x[n] is extended to 2N length by adding zero elements: 


x{[n], O<n<sN-1 
x,[n] = v7 
0, N<n<2N-1 


co anew sequence, y[n], of length 2N is created from x,[n] using y[n] = x,[n] + x,[2N - 
-—n],0<n<2N-1,and its DFT Y[K], of length 2N, is calculated by: 


2N-1 
YiK=  ylnwsi, O<k<s2N-1 


n=0 
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ARCHITECTURE OF ENCODER BASED ON DCT 
SOURCE FORWARD: ENTROPY COMPRESSED 
IMAGE : QUANTIZER eee 
DATA as Ea eres DATA 
FIGURE 5.9 


Block Diagram of a JPEG encoder. 
Then, the DCT Cx[k] of x[n] is obtained by using: 


WxZY[k]L OSkSN-1 


C,[k]= 
0, elsewhere 


(5.49) 


The DCT transform applies mostly in compression of video and audio signals. This 
transform converts the discrete signal into a series of simple integer values (the transform 
coefficients) that constitute the amplitudes of the frequencies of the original image. It is 
then easy to make the smaller coefficients zero and achieve significant compression. 

JPEG belongs to the category of transform coding techniques, i.e., techniques that com- 
press the transform of the signal and not the signal, itself. The most widely used technique 
is the DCT. The DCT’s energy-compression capability has as a result that only a few of the 
transform coefficients have significant values so that almost all of the energy is contained 
in these particular components. 

The JPEG encoder is illustrated in Figure 5.9. It consists of three blocks, the gradient-DCT 
descriptor, the quantizer and the entropy coder. 

An uncompressed monochrome digital image (color depth of 8-bit) is a discrete signal, 
ie., itis a series of values ranging from 0-255. Consider the image of N x M pixels arranged 
in a matrix that is divided into 8x8 pixel submatrices. The coefficients of this matrix are 
the 8-bit values (0-255) that represent the intensity of the pixels and constitute the input 
data in the DCT transform. By calculating the DCT of each submatrix, 64 DCT coefficients 
are generated for each one, which indicate the weighting factor of each frequency that con- 
tributes to the synthesis of the discrete image. The purpose of the quantizer is to achieve 
greater compression by representing each coefficient with no more digits than needed. 
That is, the quantizer tries to eliminate any information that is not perceived by human 
vision. After quantization, many of the DCT coefficients have been reset. For compressing 
these data, JPEG is usually combined with an entropy coding algorithm. Entropy coding 
refers to techniques that do not take into account the type of information to be compressed. 
In other words, these techniques treat information as a simple sequence of bits. That is the 
reason why entropy coding can be applied independently of the type of information. In 
addition, entropy coding techniques offer coding without losses. 


5.8 Wavelet Transform 


If a time-series is stationary, it is reasonable to assume that its harmonic components can 
be detected by means of Fourier analysis, e.g., using DFT. Nevertheless, in various practical 
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applications, it became evident that many time-series are not stationary (ie., their mean 
statistical properties change in time). The waves of infinite support that form the har- 
monic components are not adequate in such a case, whereas waves localized not only in 
frequency but in time are required, as well. These waves are generally known as wavelets 
and allow a time-scale decomposition of a signal. The Discrete Wavelet Transform (DWT) 
is used in a variety of signal processing applications, such as video compression, Internet 
communications compression, and numerical analysis. It can efficiently represent non- 
stationary signals. 

As already mentioned, the wavelet transform is discrete in time and scale. In other 
words, the DWT coefficients may have real values; yet, under the condition of integer- 
only time and scale values used to index these coefficients. This subsection discusses the 
fundamental principles of wavelet transform and shows how the wavelet breaks a signal 
down into detail signals and an approximation. 

Define a certain level of resolution as octave. Then, each octave can be encountered as 
a pair of FIR filters. One filter of the analysis (wavelet transform) pair is a LPF, while the 
other is a HPF, as illustrated in Figure 5.10. Each filter is followed by a down-sampler so 
as to enhance the transform efficiency. Figure 5.11 shows the synthesis (inverse wavelet 
transform) pair, consisting of an inverse low-pass filter (ILPF) and an inverse high-pass 
filter (IHPF), each preceded by an up-sampler. The low-pass and high-pass filters derive 
the average and detail signal, respectively. For example, a simple low-pass filter may have 
coefficients {1/2, 1/2}, producing an output (x[n] + x[n - 1])/2, which is clearly the average of 
two samples. A corresponding simple high-pass filter would have coefficients {1/2, -1/2}, 
producing an output (x[n] — x[n — 1])/2, which is half the difference of the given samples. 
While the average signal would be similar to the original one, the details are required so 
as to make the reconstructed signal match the original one. Multi-resolution analysis feeds 
the average signal into another set of filters, which produces the average and detail signals 
at the next octave. Since only the detail signals are kept as the higher octave averages can 


——+| LPF + 4} + 


Input 


x[n] 


L._. pr |_| } -_- 


FIGURE 5.10 
Analysis filters. 
—— > 7 -=+ ILPF 
Output 
yln] 
—+ t }—» THPF 
FIGURE 5.11 


Synthesis filters. 
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be recomputed during the synthesis, each of the octave’s outputs have only half the input’s 
amount of data. Thus, the wavelet representation is almost the same size as the original. 

Using the discrete wavelet transform on 1D data can be accomplished via, for example, 
the MATLAB commands dwt and idwt. Furthermore, DWT on images (or other 2D data) 
can be accomplished with the dwt2 and idwt2 functions in MATLAB. 2D transform can be 
accomplished either by applying the low- and high-pass filters along the rows of the data 
and then applying these filters along the columns of the previous results, or by applying 
four matrix convolutions—one for each low-pass/high-pass, horizontal/vertical combina- 
tion. Separability means that the 2D transform is simply an application of the 1D DWT in 
the horizontal and vertical directions. The non-separable 2D transform works ina slightly 
different manner, since it computes the transform based on a 2D sample of the input con- 
volved with a matrix, but the results are identical. This can be considered as two ways of 
reaching the same answer, since we can combine two filters into one, we can therefore 
combine vertical and horizontal operations into a matrix multiplication. 

The main difference between the 1D and higher-order dimension DWT is that a pair of 
filters operates on each channel for its corresponding dimension. 


5.8.1 Wavelet Transform Theory 


The discrete wavelet transform convolves the input by the shifts and scales of the wavelet. 
Below are variables commonly used in wavelet literature: 


° grepresents the high-pass (wavelet) filter 

¢ h represents the low-pass (scaling) filter 

e Jis the total number of octaves 

e jis the current octave (used as an index 1 <j <J) 

¢ Nis the total number of inputs 

e nis the current input (used as an index 1 <n < N) 

e Lis the width of the filter (number of taps) 

e kis the current wavelet coefficient 

¢ W,@,b) represents the continuous wavelet transform (CWT) of function f 
¢ W,(j) represents the discrete wavelet transform of function f 


¢ W(jn) represents the discrete scaling function of f, except W(0,n), which is the 
input signal. 


The continuous wavelet transform is represented by: 
W,(a,b) = } F(Bw(at + b)dt 


where f(f) is the function to analyze, yis the wavelet, and y(at + b) is the shifted and scaled 
version of the wavelet at time, b, and scale, a. An alternate form of the equation is: 


wy(s,u)= | feovsy(stt—u))at 
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again, where y is the wavelet, while the wavelet family is shown above as ,/sy(s(t- u)), 
shifted by u and scaled by s. We can rewrite the wavelet transform as an inner product: 


W,(s,u) =( f(t), Vsw(s(t—1))). 


This inner product is essentially computed by the involved filters. 

Thus far, we focused on how to get the wavelet transform given a wavelet, but how does 
one get the wavelet coefficients and implement the transform with filters? The relationship 
between wavelets and the filter banks that implement the wavelet transform is as follows. 
The scaling function, ¢(t), is determined by recursively applying the filter coefficients, 
since multi-resolution recursively convolutes the input vector after shifting and scaling. 
All the required information about the scaling and wavelet functions is obtained by the 
coefficients of the scaling function and of the wavelet function, respectively. The scaling 
function is given by: 


o(t)= V2) hlklp2t—). 
k 
The wavelet function is expressed as: 


y()=N2> stklp(2t—b. 
k 


There is a finite set of coefficients, h[k]. Once these coefficients are found, allowing us to 
design the low-pass filter, then the high-pass filter coefficients can be extracted quite easily. 
The low-pass (scaling) coefficients are: 


(es G443 Sage TayG 


4/2" 4/2 ° 4J2 7 ay2 © 


h{O], h[1], h[2], h[3] = 


These produce a space, Vi, which is invariant to shift and scale and which is required 
for multi-resolution analysis. Now that we have coefficients for the low-pass filter, the cor- 
responding high-pass (wavelet) filter coefficients can be calculated, as follows: 


1-3 -34+V3 3+ 3 -1- v3 
0], [11 2[2], , 4 
lO], gf1L [2], [3] V2 4/2" 4/2 4/2 


Notice that g[0] = h[3], g[1] = —-h[2], g[2] = hl], and g[3] = —-h[0]. This pattern is explicitly 
shown in Figure 5.12. 
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FIGURE 5.12 
A quadrature mirror filter. 


The significance of the above coefficients is that they are used in a filter bank to generate 
the wavelet transform. Writing the wavelet equation with regard to the discrete sampled 


sequence x[n] we get: 
aL+b-1 
1 n—b 
W,[a,b] = — x[n ( 
L4,b] r- 2 Inlg . 


where the wavelet is replaced by function g, which is obtained by sampling the continuous 
wavelet function. For the discrete case, we let a = 2k such that the parameters (a, b) and k 
take integer-only values. Particularly, we replace a and b with] and n. By using the discrete 
case, redundancy is not required in the transformed signal. To reduce redundancy, the 
wavelet equation must satisfy certain conditions. First, we must introduce an orthogonal 
scaling function, which allows us to have an approximation at the last level of resolution 
(the initial state is simply denoted by the input signal). Hence, the coefficients for the low- 
and high-pass filter are intimately related. The functions that produce these coefficients 
are also dependent on each other. Second, the representation of the signal at octave j must 
have all the information of the signal at octave j + 1, that is: the input signal has more infor- 
mation than the first approximation of this signal. The function x[n] is thus changed to 
WIj - 1, m], which is the scaling function’s decomposition from the previous level of resolu- 
tion, with m as an index. Third, after J levels of resolution, the result of the scaling func- 
tion on the signal will be 0. After repeatedly viewing a signal in successive approximation 
rounds, eventually the scaling function will not produce any useful information. Fourth, 
the scaling function allows us to approximate any given signal with a variable amount of 
precision. The scaling function, 1, provides an approximation of the signal via the follow- 
ing equation. This is also known as the low-pass output: 


Wipnl= S Wii —1,mJh[2n- m]. 


m=0 


The wavelet function gives us the detail signal, namely, the high-pass output: 


W,Litl = SL WLi- 1, mlgl2n - mt 


m=0 
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The n term gives us the shift, the starting points for the wavelet calculations. The index 
2n — m incorporates the scaling, resulting in half of the outputs for octave j compared to 
the previous octave j — 1. 


5.9 SOLVED PROBLEMS 


1, Osns3 


. Fi h : 
0, elsewhere ind the magnitude and 


5.9.1 Let the discrete sequence soi | 
phase of its DTFT. 


SOLUTION: The DTFT of x[n] is derived by: 


3 
X(@)= Si x(ne™ =1+e +e PO +e Be 
0 
7 Fogo _ sin(2@) oP 
1-e!° — sin(@/2) 


Thus, the magnitude of DTFT is IX(a) = cuGuy , and the phase: 
sin(@/2) 
_ 3a sin(2@) 
y . 2 
2X(o) = sin(co/ yo 
3@ sin(2@) 
=, wher ———— 
2 sin(@/2) 


2 
5.9.2 Let the discrete sequence x[n]=cosw yn where wWy= “7 Bind and plot the 
DTFT of the signal. 2 


SOLUTION: From the property X (ev) = S'215(w — Wy) — 2a ), we obtain: 


=—0o 


X(e) = ¥ 40-24 -ant} + a{ w+ 2-271). 
l=—co 


Thus, 


x(o™) =15{ w— 2% |+n9{ w+ 2 —m<w<n 
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a x(el) 


The figure above shows that X(e”) is repeated with a period 2n. 
5.9.3 Consider an LTT system. It is given that: 
h[n] = a"u[n] 


x[n] = b” u[n] 


where h[n] its impulse response and x[n], the input, respectively, and |a| <1, |b| <1. 
Find the response of the system using DTFT. 


SOLUTION: The DTFTs of h[n] and x[n] are: 


; 1 1 
x(e#) = 4 Hews. 2 
le") 1— be te") 1—ae i 


Thus, the system response will be: y[n] = IDTFT[DTFT (h[n])DTFT(x[n))]. 


Y (ci) = x(e”). H(e) = = 
(1- ae! )(1- be”) 
. A B 

ae (1- ae) : (1—be7i*) 

From the equations above we calculate A and B: 
a b 
er 7 aia Go ark) . ees 

Thus, yln]=—— [a u[n]—b""'ufn]] for o#b. 


5.9.4 Let the periodic sequence be x[n], with a period N=2. Find the Fourier coef- 
ficients of the sequence. 
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SOLUTION: The Fourier coefficients of the periodic sequence, x[n], are: 


X[k]= SY slnle | oar [0] + X[1(—1)* =1+2-(-1)', k=0,1 
Thus, X[0]=3, X[1J=-1 


and the sequence, x[n], can be written as: 
7 Iw: i iy 
i{n] = IDFS{XIK]} = YY Xlkle! 2” =1.5-0.5.(-1) 


5.9.5 Let the periodic sequence be x[n] with a period N = 10. 
a. Find the Fourier coefficients X[k] of the sequence 
b. Plot the magnitude X[K] and the phase Arg(X[k]). 


x[n] 


SOLUTION: 
N=10 


4 Raja 00,000,112, 


(1) 
i} 
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The Fourier coefficients of the sequence x[”] are: 


5 
2nk 
4 4 {2x 2nk | S| 
~ —j| — |nk es 10 
= tel= Dwg = eto ye leae) te 
n=0 n=0 n=0 1 e “10- 
: -{ 4) _ ak mk is _ atk 
1-e’™ 1-e \? ? ~ e 2e@2-e 2e ? (2) 
= 2nk = (ak xk = X[k] mk .mk mk atk 
jap etl 3) a 0s op We 
ak ak 1k eee 
e/2 ee - a) < a Bee 
=> X[k] 7 =e 10 
fa D in| 2jsin ™ sin 7 
ae 10 10 
b. The magnitude of X[k] is: 
; mk 
(3)) _,4zk sin—-| |S cs 
Pe a ica 0 
sin—| |sin—— 
10 10 
mk a wk 
7 sin — — cos — 
For k= 0= lim|X[k]| = lim|——2_| = lim| 22 =5 
k30 k30 wk} ko; w 1k 
10 10 10 
For k=1>5 lim|X[K] = 3.24 
For k=2= lim|X[k]|= 0 
For k=3= lim|X[k] = 1.24 
For k= 4= lim|X[k]]= 0 
For k=5 = lim|X(k) =1 
__ Wk 
5 5 -;Azk sin 4nk 
The phase of X[k] is: Arg| X[k] =Argle 1 ol aay (4) 
sin — 
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The magnitude and phase of the sequence, x[n], are illustrated in the following 


figure. 
magnitude 
Se 1 1 
| e 
0 , ft . f 4 Tf 4, 
ie} 2 4 6 8 10 
phase (rad) 
2 ] Z 
Oe I di t ad ad e hi ad 


5.9.6 Compute and plot the magnitude spectra of signals 


0, n<-N, 
x[n]=a"u[n], |a| <landh[n]=51, -N,<n<N, 
0, n>N, 
SOLUTION: 
a. The magnitude spectrum of the signal x[n] = a"u[n], |a| < 1 is computed as 
follows: 
— = — _ yn 1 
x(e) = ane" = Yate = Y (gei#)" = 
1 1 
=> |Xlel? = —| = 1 
(e ) 1-ae!’| |(1—acos(@))— jasin(@) @) 
7 1 
\(1—acos(w))? +(asin(@))? 
NOTE THAT: 
(1-acos(@))? +(asin(@)) = 1—2acos(@) + a? cos?(@) + a’ sin’ (@) 
= 1-2acos(@)+a’ 2) 
Consequently, 
: 1 
x (el ) = (3) 


a —2acos(w) +a? 
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x(e)] = : = : 
| : ja=0 J1-2acos(0) +a? V(1-2a+a?) 
(4) 
ee eee 
(1-a) 1-4 
x(ci) : : 
| ' o=n J1-2acos(r) +0” V(1+2a+0?) 
(5) 


1 1 


(l1+ay? +a 


The magnitude spectrum of the signal is plotted in the figures below. Note that 
for a > 0, we have a low-pass filter (LPF), while for a < 0, we have a high-pass filter 


(HPF). 


Ix(e")I a>0 


1/(1-a) 


jw 
Ix(e")| a0 
1/(1+a)} HPF 
1/(1-a) 
-2n TT 0 me an w 


0, n<-N, 
b. The magnitude spectrum of the signal h[n]=; 1, -N,<n<N, is com- 
puted as follows: 0 n>N, 


») jon YS (joy? __ sin(@(N, +1/ 2) 
Heh) Dea Dyer) = sin(/2) ®) 


n=—-Ny n=—Ny 
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The sequence h[n] for N, = 2 is plotted as follows 


x(n] 


The function H(e!*) is very similar to sin c = sin(x)/x and is periodic with a 
period 2n. 


H(e”) 


5 |. 2Ni+1 ee 


5.9.7 Plot the energy spectral density of the signal x[n] = a"u[n], |a| < 1 for a=0.7 and 
a =-0.7. 


SOLUTION: The signal spectrum is 


co co co 


X(ei#)= SY xe" = ae ion =) (aei#)" = 1 


1-ae! 


n=—co n=0 n=0 


Thus, the energy spectral density is 


(Af =X(NX()=2 L a : 
—a 


e PF 1—ae®*! 1 —2acos(2nf) +a 


In the figures below the energy spectral density for a = 0.7 and for a = -0.7 is 
illustrated. Note that for a = 0.7, the signal has stronger low frequencies, while for 
a =-0.7, the signal has stronger high frequencies. This can be explained as follows: 
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The signal x[n] = a"u[n] converges exponentially to zero as n >0o. However, if « is 
negative, for example a = —0.7, this convergence occurs with alternate positive and 
negative values, which means that the signal exhibits sharp changes and, conse- 
quently, a strong high frequency spectral content. 


-0.5 -0.4 -0.3 -0.2 -0.1 0 01 02 03 04 0.5 


-0.5 -0.4 -0.3 -0.2 -0.1 0 01 02 03 04 0.5 


5.9.8 Let the discrete-time signal: 


L|n|sk 
x[n] = 
0,|n|>k 
a. Compute the DFT of N length of the signal x[n], where N > 2k + 1. 
Plot the magnitude and phase of the DFT when k = 2 and N = 12. 


Estimate the energy of the signal contained between f, = 1kHz and f, = 
3kHz, if the sampling frequency f, is 8kHz. 
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SOLUTION: 
a. From the signal, it is derived: 


1, |n|sk 1, Os<ns<2k 
x[n]= => x[n-k]= 
0, |nb>k 0, elsewhere 
It is true that: 
ee 
x[n— kJ] w""X[m], we 7X 
N-1 2k 
=> X[m] — w"Eper {x[n - ky} > DFT {x{n - ky} = Sworn -kj= >a" 
n=0 n=0 
1 1 1 
(2k+1)m _ (3)| Gaz _ - me] 
= DFT {x{n-K]}= 2+ * = xfmj=w "2 
oe aes 
2j-sin| (e+) sn] (2k-+7— 
= X[m]= N J_ N J m=0,1,2,...(N—1) 
.. { wm ™ 
2]: sn) sin 
N N 


Another way of calculating X[m] is: 


xim= wt x[n 1= Yum = wr +..414+..40™ 
n=—-k 


n=—k 
(2k+1)m 
— mk W -1 
=w™(1+..4w™) => X[m] =0-™ ——___— 


w"-1 

b. Fork =2 «aN = 12, we have: 
. 5am 

Ss peibinaaanindaaeer 

X[m] = ——12_,m=0,1,2,...(N—D). 

mm 

sin| —— 
( 12 


It is true that: | X[0] = X[12]|, | X[1] = X[11]], |X[2] = X[10]] ..., etc. 
Generally, it is: | X[m] = X[N - m]|. 
After some calculations, the following is obtained: 


X[0]=5, X[1] = 3.73, X[2] = 1, X[3] =-1, X[4] = -1, X[5] = 0.268, and X[6]= 
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Knowing that the magnitude of the DFT will have an even symmetry with 


respect to the point = =6, and the phase will have an odd symmetry with 


respect to the same point, we plot the magnitude and phase spectra of the DFT 
of the signal below. 


| xt) 
7 
3.73 
: Hasna " 
| ArelX(m) 
: . 
o Bis R= fle _1 fi, _ 8kHz _ 3 


fs 8kHz 8’ * f, S8kHz 8° 


Q, = 2nF, = 20 = © and Q = 2nF, = 20> = a 


8 4° 
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From the relations above, it is: Q; <Q <Q,>2<m <4, hence, the energy of the 
signal between the frequencies f, 1kHz and f, 3kHz is: 


e=12> xem! ==(7+14+1)= 


m=2 


- 
a 


5.9.9 a. Show that the DFT of the unit pulse, 6 [1], is a constant, which is indepen- 
dent of the number of zeros that follow the pulse within the analysis win- 
dow of length N. 


b. Show that the DFT of a constant signal contains only one frequency within 
the analysis window (ie., x,[n] = A for0 <n < N-1). 


SOLUTION: 
a. After applying the definition of the DFT, we get: 
N-1 2mn-m 2am 
ea [ae 9 
x,[n] = 6[n] => X,[m] = DFT[6[n]] = > an -e N =1-e N =1. 
n=0 


which is true for Vm € [0, N — 1] and VN > 1. 


b. For the constant signal, x,[n] =A for0 <n < N-1, itis x,[n] = A [u[n] — u[n - N]]. 
The DFT of the signal is derived as follows: 


X,[m] = DET[ A(u[n]-ufn—N]) = ay ew SA 
n=0 


The equation above derives from: 


y eee | 0, Vm#0 
a N, m=0 

The duality in the above cases is easy to explain from a physical point of 
view if we consider the corresponding analog signals. Indeed, the signal, 6 (t), 
contains all frequencies, with constant amplitude and, respectively, the signal 
8[n] gives a constant signal for all m values in the analysis window, [0, N — 1]. 
Respectively, the continuous (constant) signal contains a single frequency: 
zero. Similarly, the constant signal in the analysis window, [0, N — 1], gives a 
non-zero value only for m = 0. 


5.9.10 Compute the DFT of the sequence x[n] = {1, 0, 0, 1}. The IDFT of the sequence 
X(k) will be evaluated in order to verify the result. 


SOLUTION: x(0] = 1, x[T] = 0, x[2T] = 0, x[3T] =1,N=4 


3 3 
k=0> X[0]= yal nT]e” = Yan] + XT] + 12T] + 2137] 
n=0 n=0 


=1+0+0+1=2 
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3 
k=1>X[\J= > x[nTle"" 
n=0 
_ 6% 
x[nT]e?"/" =1404041e 4 =14] 


3 
n=0 


k=2=>X[2]= ¥ x(nT]e i?" =14+04+04+ le" =1-1=0 


3 
n=0 


3 On 
k=3= X[3]= ) xn T]e PP" =140+0+1e |? =1-j 
n=0 
Thus, 
X[k] ={2,(1+ j),0,1-j)} 
We can study the result be computing x[nT] 
iw 1 
n=0=>x{0]= fo X[k]= qL X10 + X[1]+ X[2]+ X[3] | 
k=0 


=i[2+14j+0+1-j]=1 


n=1>2[T]=0 


n=2=> x[2T]=0 


isn 
n=3=>  x[3]= X[kle**” = asa pe? +(1- pore | 


k=0 


=i2+a+ncaea-ap]=1 


Thus, 


anT]={1 0 0 1} 
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2ak. 
5.9.11 Compute the DFT of length, N, of the signal x[n] = cos( as "0 <nsN-1, 
where k is an integer in the interval 0 to N-1. 


SOLUTION: We will use Euler’s formula: 


.2mkn .2mkn .2mkn .2n(N-k)n 
2nkn) 1 (7 1 5 1; 1i 

=~e N +-e N =-e N+-e N 
N 2 2 


x[n] = cos( 


N 


2nkn 2n(N-k 
1|N i N jee 
=—|—e N +—e : 


From the equation above and the definition of the inverse DFT, 


N-1 2mm 
x[n] = ~ X[mle’ N_, the following is obtained: 
m=0 


Ny TS isan. 
0, m#k 


For k = 0 or k = N/2 (if N is even): ximi=| 
. m=k 


m, 
2 
Otherwise: X[m]= a8 m= Na 
2 
0 


: elsewhere 


5.9.12 The DFT X[k] of length, N, of the signal x[n], 0 < n < N — 1 is considered 


known. Compute the DFTs of length N of the signals x,[n]= x[n]cos anim 


x,[n] = x[n]sin ae, 0<n<N-1 where k is an integer in the interval 0 to N-1. 


and 


SOLUTION: This is a case of amplitude modulation (AM) leading in a frequency 
shift. 


According to Euler’s formula: 


eN+eN 1 jam _ >a 
x,[n] = x[n] =—x[nJe N +—x[nJe %. 


N 
N 


Making use of the linearity and shifting properties of the DFT in the frequency 
domain we have: 


X [m]= >Xitm Ky +5 Ul Ally, O0<m<N-1, 
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where (())y denotes modulo N. In the same way, we find: 


X,{ml= XI ky Sit Ello, O<mEN=1. 
j 2] 


5.9.13 Let the sequence x[n]= {h 2, 3, 4}. Illustrate the sequences x[(n — 1)mod 4], 
x[(1 + 1)mod 4] and x[-n)mod 4]. 


SOLUTION: The sequence x[1] is illustrated in the following figure. 


4 


x[n]= {1,2,3,4} 
3 1 


The sequence x[(n — 1)mod 4], ie., a circular clockwise shift of the sequence x[n] 
is illustrated below: 


3 
x[(n-1)mod 4] = {4,1,2,3} 
2 4 
1 


The sequence x[(n + 1)mod 4], i.e., a circular anticlockwise shift of the sequence 
x[n] is illustrated below: 


1 
x[(n+1)mod 4] = {2,3,4,1} 
4 2 
3 


Let’s assume that the sequence x[1] is periodic with a period, N (here N = 4), that 
is, x[n] = x[n + N], and it is illustrated in the following figure. 
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x[0]=1 

x[-1] = x[-1+ 4] = x[3] =4 
x[-2] = x[-2 + 4] = x[2]=3 
x[-3] = x[-3 + 4] = 2[1] =2 


Thus, 


x{-n] = {x{0], x{-1], x[-2], x{-3]} = x{-n] = {14 3,21 


The sequence x[(—)mod 4] is illustrated below: 


x[-n mod 4] = {1,4,3,2} 
3 il 


5.9.14 Compute the 4-point circular convolution for the signals x,[n] = {1, 2, 2, 0} and 
x,[n] = {1, 2, 3, 4}. 


SOLUTION: 


x,[n]= {1,2,2,0} and x,[n]= {1,2,3,4} 
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Forn=0 


3 3 3: 
Y' xilmx,[10— ml], = }°[{1,2,2,0} -{1,4,3,2}] > {1,8,6,0} =15 


m=0 m=0 m=0 


Forn=1 


3 3 3 
SY xilm}xs[0- ml], = ¥°[{2,2,2,0} -{2,1,4,3} ] > {2,2,8,0} = 12 


m=0 m=0 m=0 


xl l0— ml ml .-¥[b, 2,2,0}-{3,2,1,4}| ING. 4,2,0}=9 


m=0 m= m=0 


Forn=3 


¥ xltl0—m n= > [t 2,2,0}-{4,3,2,1} | Di 6,4,0}=14 


m=0 m=0 m=0 


Thus, x,[n] @,x,[n] ={15, 12, 9, 14} 

We can easily verify ree 

DFT (length 4) of x,[n] = {5, -1 — 2, 1, -1 + j2} 
DFT (length 4) of x,[n] = | —2 + j2,2,-2 -j2} 
and DFT [x,[n]]-DFT [x,[n]] = {50, 6+ j2, -2, 6 — j2} 
Finally: IDFT {50, 6+ j2, -2, 6 — j2} = {15, 12, 9, 14} 


For the previous example, if we get a 3 + 4-1 = 6-point circular convolution 
(length (x1) = 3), we will have x,[n] @, x,[n] = {1, 4, 9, 14, 14, 8}, which is the same as 
the result of the linear convolution: conv(x1, x2) = 149 14 148. 


5.9.15 Let the sequences x[n]= {12,3} and h[n]= {12} ; 
a. Compute the circular convolution of the sequences. 
b. Compute the linear convolution of the sequences using the circular convolution. 
c. Compute the linear convolution of the sequences using the DFT. 


SOLUTION: 
a. The circular convolution of x[n] and h[n] is: 


fln] =x[n]® h[n] = x([n- k] mod 3) h[k] (1) 


2 
k=0 
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We will implement relation (1) by placing the samples of the sequences /i[k] = 
{1, 2, 0} and x[-A] = {1, 3, 2} in two concentric circles, as shown in the figures below. 


2 


% 


h{k] 1 


x[-k] 


It is: f[0] = (1, 2, 0) x (1,3, 2)? =10142+3=7 


3 


\ 


h[k] 2 


It is: f[1] = (1, 2,0) x 2, 1,3)? =1-¢24+201=4 


uf 


% 


h{k] 3 


x[2-k] 2 


It is: f[2] = (1, 2, 0) x G, 2, 1)7=1+34+202=7 
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Consequently, the circular convolution of x[] and h[n]is f[n]= 7,4, 7| (2) 


The length of the sequence x[n]= {1, 2,3} is N, = 3, whereas the length of 


the sequence hi[n] = {1, 2} is N, = 2. The linear convolution of x[n] and h[n] 
creates a sequence of length N, + N, - 1 =3 + 2-1 = 4. Consequently, we 
must add one and two zeros to x[n] and h[n] sequences, respectively, ie., 


x[n]= {L 2,3,0} and h[n]= {L 2,0, 0}, 


The circular convolution, x[n] ® h[n], will be: 
3 
y[n] = x[n] © h[n] = SY lieder —k)mod 4] (3) 


Thus, 


ee eal 
T=1-24+2-1=4 
T=1-34+2-2=7 
T=2-3=6 


y[0] = (1,2,0,0) x (1,0,3,2 
y[1] = (1,2,0,0) x (2,1,0,3 
y[2] = (1,2,0,0) x (3,2,1,0 


[ 
[ 
[ 
y[3] = (1,2,0,0) x (0,3,2,1 


Using the circular convolution, the linear convolution of the sequences is 
y[n]= {14 7,61, The result is the same with that of the linear convolution of 
the sequences x[n] and h[n]. 


In order to calculate the linear convolution of x[n] and h[n], first we will add the 
appropriate number of zeros to the sequences so that their length is N = N, + 
N,-1=4, that is x[n] = {1, 2, 3, 0} and y[n] = {1, 2, 0, O}. 


3 
The DFT of x[n] is: X[k] = DFT [ x{n]]= SY alee (5) 


n=0 


Consequently, the DFT matrix for x[n] will be: 


X[0] We We We We |} x{0] 
XT) |_| We We We We |] xf 
Xi2]| | wo w2 we we |} 212] (6) 
X[3] [3] 


W:; Wr Wr Wy |L* 


where: 


on \9 
Wiles) =1 


20 4 


la eh uT .. 
Wise * Se * = cos, — jsin 


so 
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20 
ess a3 2 
Wp=e 4 =e" =cosa-jsinnt=-1 


2 
n Wr=(Wi) ==? =-1 
We = We ‘Wi = ik-)) = ; 


wi =(w?) =(»?=1=W? 


Thus, 
X[0] 1% 4&4 4 IIa 6 
Al | )2 =F =f J lhe) ez] 
X[2] ee es ames oe Dec a De 
X[3] 1 ¢ =<? file 2+2j 

That is, X[k] = DFT[x[n]] = {6,-2-2j,-2+2/} 

Likewise, for H[k]: 

H[0] 1 1 1 =#14fy 3 
+16 Re et a! (2 UV | 
H[2] 1 4 2 -=t//a)">)] 4 
H[3] 1 sf = =7)/0 1+2; 


That is, H[k]= DFT[h[n]]={3, 1-27, -1, 14+2]} 
It is true that: 
Y[k] = DFT[y[n]] = X[KJH[k] => 
Y(kK)={6, —2-2j, 2, -2+2j}«{3, 1-2), -1,1+2j}> 


Y[k]={18, -6+2j, -2, -6—-2)) 
But y[n] = IDFT | Y[k]]= 


So, the IDFT matrix for y[n] will be: 


[0] [ 
[1] [ 
yl2]}| 4) wo we? we Ww? Y[2] 
y[3] we we, Ww, w,t | | YDS) 


W? W We W; Y 
1| We Wr We WP ¥ 


y 


(10) 


(11) 


(12) 


(13) 
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where 


255 


ee (14) 


ts 

— 

| 

— 

NS) 
DN FR 


Thus, y[n] = {1, 4, 7, 6}. The result is the same with that of the linear convolu- 
tion of x[n] and h[n]. 


5.9.16 Let {g[n]} = {g[n]} = (5, 2, 4, -1, 0, O} and h[n] = {-3, 4, 0, 2, -1, 2}, two sequences 


of finite length. 


a. Compute the circular convolution y_[n] = g,[n] * .h[n]. 
b. Compute the circular convolution using the DFT. 


5 


SOLUTION: The circular convolution y,[n] = g,[n}*,h[n] is: yc] = SY" selkHl(n —k))e1, 
where {g,[1]} = {5, 2, 4, -1, 0, O}. k=0 


yl0] = gLOIM[0]+ g.[1A[5] + g [214] + g[3]13] + ¢ [4]h12] 
+g [5]h[1] = glO]A[0] + gfJn[5] + g[2}h[4]+ g[3]n[3] 
=5x(-3)+2x2+4x(-1)+(-1) x2 =-17 


yc] = gLOVMIU + g.[Mh10] + g.[2][5] + g.[3]n[4] + ¢.[4]h[3] 
+8 [5]h[2] = glO}[1] + gf1]n[0]+ g[2]A[5] + g[3]A[4] 
=5x44+2x(-3)+4x2+(-1)x(-1) = 23 


ycl2]= g.[0]M[2] + g [UA] + g.[2}h[0] + g.[3]n[5]+ g. [4114] 
+8 [5]A[3] = glO}n[2]+ g{1A{1]+ gl2]n[0]+ g[3]A15] 
=5x04+2x4+4x(-3)+(-1)x2=-6 
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ycl3] = glO\M3]+ g-LAl2]+ g [2] + g.[3]M[0]+ g-[4]h15] 
+ e[5)h[4] = glO}n[3] + gl1]h[2]+ gl2 a [1] + g[3]h[0] 
=5x2+2x0+4x4+(-1)x(-3)= 


ycl4] = g.[0]M[4] + g [1 A3] + g-[2)M[2]+ g-[3)A[1] + g.[4]h10] 
+8 [5]h[5] = glO}n[4]+ gllh[3] + g[2}h[2]+ g[3]h[1] 
=5x(-1)+2x24+4x04+(-1)x4=-5 


¥cl5] = g LOIM[5]+ g.[UAl4] + g [2113] + g[3]l2] + g [4] 
+g [5]h[0] = glO}n[5] + glh[4] + gl2}h[3] + gl3}h[2] 
=5x24+2x(-1)+4x2+(-1)x0=16 


b. Next, the two DFTs (length 6) are derived: 


.2mkn ei 


ye g.[nle’ & and H{k =n h[nje © , resulting to: 


n=0 


{G,[K}} = (10, 5 — 5.1962, 1 + — jl. "7322, 8, 1 - j1.7322, 5 + j5.1962} and 
{H[k]} = {4,-1.5 — j2.5981,-3.5 — j0.866,-12,—3.5 + j0.866,-1.5 + j2.5981} 


Thus, 


{G,[k]H[k]} = {40,-21 — j5.1962,-2 — j6.9282,-96,-2 + 


e 


+6.9282,—21+ j5.1962} 


The circular os can be computed by the inverse DFT of the product 
a 


above, ie, yo[n PS eune , providing the following result: 


{y-[n]} = {-17, 23, -6, 09, —5, 16}, which is the same with the one estimated in (a). 


5.9.17 Derive the DFT of a 4-point sequence using the radix-2 FFT algorithm when: 
a. x[n] = {x[0], x[1], x[2], x[3]} and 
b. x[n] = {1, 11, 1}. 


SOLUTION: 
a. N=4=2*>0=2. Thus, there are two segmentation stages. The DFT is: 


3 
X[k] = DET [ax{n]]=¥ xinwy", k=0,1,2,3 > 


n=0 
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X[k] = x[O]W? + x[1]We + x[2]w?* +. x[3]We*, k=0,1,2,3 (1) 


Then, from Eq. (1) we can conclude that in order to calculate X[k], 42=16 com- 
plex multiplications are needed. 


xn] ={x{0], x{2]} 


B titi f [ L t 
a Ae ae eee 


The DFT of x,[n] and x,[n] is: 


X[k] =DET [x,[n]]= > x.[nlW2" = x,[0W? +x,1Ws = 5 
n=0 


X[k] = x[O]W? + x[1]Ws, k=0,1 


£ 


Likewise, X,[k]= DFT | x,[1] ]=x{1]W3'+x13]Ws, k=0,1 


(4) 
By writing equations (3) and (4) in a general form, we obtain: 
X[k] = x[O]We, + x[2]Wx, = x[O]WY + x[2]We*, k =0,1, Broeraem 1 (5) 
2 a 
0 k 0 2k N 
and X[k] = x[1]Wy + 2x[3]Wy = x[1]Wy + x[3]Wa, k =0, a ae 1 (6) 
2 2 
From (1), obviously it is: 
X[k] = X,[k]+WyX,[k] k=0,1,2,...,N-1 (7) 
And for the remaining part of the DFT, it is: 
N k 
x] kN |= xte+21= X[k) WEX k=0,1 (8) 
where 
X [0] = x[0]+ x[2] (9) 
agen 
X [1] = x[0]+ x[2]W, = x[0]+x[2]e 7 => (10) 


X,[1] = x[0]+ x[2] e* = x[0] — x[2] 
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X [0] = x{1] + x13] (11) 
X [0] = x{1] - x13] (12) 


b. x[n] = {1,1,1, 1), N=4 
In this case, it is: x[0] = x[1] = x[2] = x[3] = 1 


(9) => X,[0]=2 
(10) > X,[1]=0 
(11) = X,[0]=2 (13) 
(12) > X,[1]=0 
k=0 
(7) = X[0]= X,[0]+ X,[0] =4 
k=1 


20 


X[J=X[lJ+e?! X, [=X [+e 2x, [> 
X[1]= X,[0]+ jX,[1] = 0 


(8) => X[2]= X,[0] + W?X,[0] = X,[0]— X,[0] = 0 


(8) => X[3]= X,[1] + W;X,[1] = X.[1]- jX,[1]=0 


Hence, the DFT of x[n], using radix-2 FFT, is X{k} = {4, 0, 0, 0} 
5.9.18 Let the real signal x[n], n = 0, 1, 2, ..., 7: 
x[n] = {2,0,1, 0,-1, 1,-2, 1} 


Compute the DFT of the signal using the FFT. How many complex mul- 
tiplications are required for the computation of the DFT and of the FFT, 
respectively? 


SOLUTION: x[n] = {2, 0, 1, 0, -1, 1, -2, 1, N=8 > v =3, so three stages of segmenta- 
tion are required. 
Ist stage 2nd stage 


Frequency Domain Analysis 


For the first segmentation stage, the FFT equations are: 


X[k] = X[k] + WEX [Kk], b= O12 poor 
Xfk-+ 1= X[I-WaXIEL b= O12, 
(1) => X[k] = X,[k] + We X[k], k=0,1,2,3 

(2) > X[k+4]=X,[k]-WEX[k], k=0,1,2,3 


The DFT of x,[n] is: 


X,[0] 1 1 1 1 x,[0] 
XL} [1 We We We] | ef) 
X[2]}] |1 We we We | | x2] 
X.[3] 1 Wr Wi We | | x13) 
X,[0] 1 ai 4 2 0 
> 0 A fe A le a (2 ey 
Sh) ha Se a a 2 
X, [3] 1 j -1 -j —2 343] 
Likewise, for X,(k), it is: 
X,[0] i 1 st 4 0 2 
AAU |b =f =. g Oy) | eg 
x12) 10 4 4 atl lal} @ 
X,[3] yf = ey == 7 
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(1) 


(2) 


(1)’ 
(2)’ 


3 
X[k] = Yi x law?" = x,[0]+x,[1]Wi + x,[2]W?* + x,[3]We* or 
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X[1]=3-3j+e /#(-14 f)=3- 1.586 
X[2[=24e 2 02 


3m 
X[3]=3+3j+e (4 (-1-j)=3+j4.414 


For the remaining part of the DFT, it is: 


(2! => X[4]= X.[0]- X,[0]=-2 


e 0 
20 


1]=3-3j+e /8 (-1+ j)=3-j4.414 


Ras 
pel 
I 
os 
= 
l 
= 
>< 


Note that if we used the second segmentation stage, the FFT equations 
would be: 


X[k] = X.o[k] + Wy Xeolk] 


N 
X,| k+— |= X[k] Wx. [k 
| +N elk] N col k] N 


k=0,1,2,...,—- 
X[k] = Xoelk] + Wy X ook] 4 


x,e+ 8 =x ,0d-Wex, (4 


The computation of the DFT requires 8’=64 complex multiplications, 
whereas the FFT requires as 


— 12 complex multiplications. 


5.9.19 Calculate the DFT of the sequence x[n] = {a, 2a, 4a, 8a} with radix-2 FFT and 


describe the process rigorously. How many multiplications are needed for the 
DFT and the FFT, respectively? 


SOLUTION: x{[n] = {a, 2a, 4a, 8a}, N = 4 = 2? = v = 2, so two stages of analysis are 
required. 


={a,4 
is aa a - ne i 
0 a y 
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FFT Equations 


X[k]= X,[k]+ WX [k] k=0,1 (1) 
X[k +2]=X,[k]-W5X,[k] k=0,1 (2) 


| X[0] = 50+ 10a = 15a 
a= 
X[1] = X,[1]+ W)X,[1] = 3a + joc 


(2) X[2] = X,[0]— W?X,[0] = 5a — 10a =-5a 
X[3] = X,[1]- W; X,[1] = X,[1]+ jX[1] = 3a — joer 


Thus, X[k] = {15a, —3a + j6a, —5a, —30 —j6a}. 
The same result can be obtained if we divide again X,[k] and X,[k] The DFT 
requires 16 multiplications, whereas the FFT requires 4 multiplications. 


5.9.20 Compute the linear convolution of f[n] and x[n] using the overlap and add 
method, when h[n] = {1, 1, 1} and x[n] = {0, 1, 2, 3,...} 


SOLUTION: The length of [1] is N=3, whereas x[n] is much longer. By choosing M=23=8, 
h[n] expands to a sequence of length 8 (by adding 5 (= M-N) zero samples at its end), ie., 


A[n] = {1,1,1,0,0,0,0,0,} (1) 
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The sequence x(n) is divided in segments of length M-N+1=8-3+1=6, and 
then, N-1=3-1=2 zeros are added at the end of each subsequence so that each of 
them becomes of length 8. Hence: 


xoln] = {0, 1, 2, 3, 4, 5, 0, O} 
x,[n] = {6,7,8,9,10,11,0,0} 
x[n] = {12,13,14, 15, 16,17,0,0} 


The circular convolution of each of the subsequences with h[n] is: 


Yoln] = xo[n] © h[n] = {0,1,2,3,4,5,0,0} @ {1,1,1,0,0,0,0,0} > 
Yo[n] = {1,1,1,0,0,0,0,0} @ {0,1,2,3,4,5,0, 0} 


X[—n] = {0,0,0,5, 4, 3,2, 1} 


thus: 


¥o[0] =(1,1,1,0,0,0,0,0) x (0,0,0,5,4,3,2,1)' =0 
¥l1] =(1,1,1,0,0,0,0,0)% (1,0,0,0,5,4,3,2) =1%1=1 
Yo(2] = (1,1,1,0,0,0,0,0) x (2,1,0,0,0,5,4,3)' =1x24+1x1=3 
YoL3] = (1,1,1,0,0,0,0,0) x (3,2,1,0,0,0,5,4)' =1x3+1x2+1x1=6 
Yo[4] = (1,1,1,0,0,0,0,0) x (4,3,2,1,0,0,0,5)" =1x4+1x3+1x2=9 
Yo15] = (1,1,1,0,0,0,0,0) x (5,4,3,2,1,0,0,0)' =1x54+1x4+1x3=12 
Yo[6] = (1,1,1,0,0,0,0,0) x (0,5,4,3,2,1,0,0) =1x0+1x5+1x4=9 
Yo[7] = (1,1,1,0,0,0,0,0) x (0,0,5,4,3,2,1,0)' =1x0+1x0+1x5 


Yoln] a ie 1,3,6, 9, 12, 9,5 


28 | last samples 


Frequency Domain Analysis 263 


Likewise, y,[n] = x,[n] ® h[n] = h[n] ® x,[n]> 


y,[n] = {1,1,1,0,0,0,0,0} © {6,7,8,9,10, 11,0, 0} 
x,[-n] = {6, 0, 0, 11, 10, 12; 8, 7} 


Likewise, y,[1]= 16 13,21, 24,27, 30,21, ui} (4) 
N-1 


last samples 


and y,[n] = x,[n] ® h[n] = {1, 1, 1, 0, 0, 0, 0, 0} @ {12, 13, 14, 15, 16, 17, 0, 0} > 
y,[n] = pee 39, 42,45, 48, 0 (5) ete. 
N-1 
The sequence of the linear convolution, y[n], consists of the above subse- 
quences, y[n], y,[n], y2[n], etc., with the difference that the last N-1=3-1=2 sam- 
ples of each segment must be added to the next N-1=2 first samples of the next 
segment. Therefore: 


y[n] = {0,1,3,6,9,12,9 + 6,5 +13,21,24,27,30,21+12,11+25,39, 42,45,..J}> 
y[n] = {0,1,3,6,9,12, 15,18, 21,24,27,30,33, 36,39, 42,45,...}. (6) 


5.9.21 Compute the linear convolution of h[n] and x[n] using the overlap and save 
method, when: h[n] = {1,1,1} and x[n] = {0,1,2,3,...} 


SOLUTION: Since, now, N=3 and M=23=8, h[n] is extended to a sequence of length 8 
by adding M-N=5 zero samples at the end, i.e., 


h[n] = {1,1,1,0,0,0,0,0} (1) 


x[n] is divided in segments of length 8 (=M) overlapped by N-1=3-1=2 sam- 
ples, i.e., 


Xo[n] = 10,0,0, 1 2, 3, 4, 5} 
x,[n] = {4,5,6,7,8,9, 10,11} 


x[n] = {10,11,12,13,14, 15,16,17} 


The circular convolution of each segment of x[n] with h[n] is: 


[n] = x,[n] © h[n] = hn] @ x,[n] => 
[n] = {1,1,1,0,0,0,0,0} @ {0,0,0,1,2,3,,4,5} 
X [—n] = {0, 5; 4, 3,2, 1, 0, O} 


Yo 
Yo 
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= y,[0] = (1,1,1,0,0,0,0,0) x (0,5,4,3,2,1,0,0)' =1x5+1x4=9 
Yol1] = (1,1,1,0,0,0,0,0) x (0,0,5,4,3,2,1,0)' =1x5=5 
¥12/=(1,1,1, 0,0,0,0,0) x(0,0,0,5,4,3,2, 17 =0 
Yo13] =(1,1,1,0,0,0,0,0) x (1,0,0,0,5,4,3,2)' =1x1=1 
Yo[4] = (1,1,1,0,0,0,0,0) x (2,1,0,0,0,5,4,3)' =1x2+1x1=3 
YoL9] = (1,1,1,0,0,0,0,0)x (3,2,1,0,0,0,5,4)’ =1x3+1x24+1x1=6 
Yo[6] = (1,1,1,0,0,0,0,0) x (4,3,2,1,0,0,0,5)' =1x4+1x3+1x2=9 


yo[7] = (1,1,1,0,0,0,0,0) x (5,4,3,2,1,0,0,0)' =1x5+1x441x3=12 


Thus, 

yo[n] = 19,5, 0,1,3,6,9, 12} (3) 
Likewise, y,[n]= x,[n] © h[n]= (25,12, 15,18,21,24, 27,30} (4) 
and y,(1) = x,(n) @ h(n) = (43, 38,33, 36,39,42,45,48} (5) 


The linear convolution sequence required is derived from all the above 
sequences provided that previously the first N-1=2 elements of each segment 
are eliminated, i.e., 


yln] = {0,1,3,6,9,12, 15,18, 21,24, 27,30,33,36,39,42,45,48,...} (6) 


We notice that the result of y[n] is the same with what we found by applying 
the overlap and add method. 


5.9.22 Let the signal cos(2x20t) be defined for n = 0:30 and sampled with a sampling 
frequency, f, = 200Hz. Plot its DIFT from —f, to f,. Do the same for the signal 
x[n] = (—0.9)" defined for n=—5:5 and for 100 frequencies in the interval 0 — 1, 
but in total from —2z to 2n. 


SOLUTION: 
a. The discrete-time signal is: 


cos(2720nT,) = co 207 = cos(270.1n) 


Ss 
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The MATLAB code for the computation of the DTFT of the signal is: 
clear; 


M=100; % M frequencies in the interval 0 - 2pi 

Fs = 200; 

n=0:30; 

k=-100:100; % (k=100 is equal to w=2*pi) 
x=cos(0.2.*pi.*n); % vector with 31 values, since vector n 
has 31 points 

X = x * (exp(-j*2*pi/M)).*( n’*k); 

F= (2*(Fs/2)/M)*k; % since fs/2 corresponds to pi, the freq. 
step size 2(Fs/2)/M 

% is a vector of length equal to the length of k ! 

plot(F, abs (X)) 

axis([-200 200 0 18]) 

xlabel('Frequency (Hz) ') 


ylabel('Magnitude of X, |X|') 


=, 
oa 
T 


Magnitude of X, |X| 
3 


oa 
T 


0 1 i 1 
-200 -100 0 100 200 
Frequency (Hz) 


We can easily observe that the signal has two components at 20 Hz and 20 
Hz, as expected. Also, the spectrum is periodic with frequency fs = 200 Hz. 


b. The MATLAB code for the computation of the DTFT of the signal is: 
clear; 


M=100; 
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n=-5:5; 

k=-200:200; %& (k=200 is equal to w=2*pi since k=100 is w=pi) 
x=(-0.9).“*n; 

X = x * (exp(-j*pi/M)).*( n’*k); 

W= (pi/M) *k; 

plot (W/pi, abs(X)) % (We avoid a and its multiples at the 
horizontal axis!) 


axis([-2 2 0 15]) 


5.9.23 Plot the DIFT of the signal x[n] = 0.8", 0 <n < 20 at the frequency range —5n < Q 
< 5a. 


SOLUTION: Since the DTFT of a signal is a complex function, we shall plot its magni- 
tude and phase. The MATLAB code for the computation of the DTFT of the signal is: 
% Generation of the signal x[n] = 0.8", 0 <n < 20. 
syms w 
n= 0:20; 
x=0.8.%n; 
% Computation of the DTFT of x[n] 
X=sum(x.*exp(-j*w*n) ); 


ezplot(abs(X), [-5*pi 5*pil); 


legend(‘Magnitude of the DTFT at 5 periods’) 
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Magnitude of the DTFT at 5 periods 


It is evident that the DTFT is periodic (at the frequency domain) with a 
period of 2x. 


>>% Plot of the phase of the DTFT at -5a < Q < 5a 
wl=-5*pi:.01:5*pi; 

XX=subs (X,w,wl) ; 

plot (w1,angle (XX) ) ; 

xlim([-5*pi 5*pil]) 

legend(‘Phase of the DTFT at 5 periods’ ) 


5.9.24 Compute the DFT of the sequence x[n] = [1, 2, 2, 1], 0 <n < 3 Plot the magni- 
tude, the phase, the real and the imaginary part of the DFT X, of the signal. 


SOLUTION: We compute the DFT of the sequence x[n] according to the following 
ae iBank 
relation X, = ¥' x(nbe MeO Aa Na, 


n=0 
x=[1 2 2 1]; 

N=length (x) ; 

for k=0:N-1 

for n=0:N-1 

X (n+1) =x (n+1) *exp (-j*2*pi*k*n/N) ; 
end 

Xk (k+1) =sum(X) ; 

end 


Xk 
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The DFT of the sequence x[n] = [1, 2, 2, 1],0 <n 3 is: X,= {6,-—1—1,0, -1 + i}. 


Plot of the magnitude |X,| of 


Q 


stem(0:N-1,magnitude) ; 
legend ('Magnitude of Xk') 
xlim([-.5 3.5]); 


ylim([-.5 6.5]); 


% magnitude=abs (Xk) ; 


the DFT X, 


6r 


5r 


—® Magnitude of Xk|- 


1 


SPlot of the phase 2X, of the 


Phase=angle (Xk) ; 


stem(0:N-1,Phase) ; 


legend ('Phase Xk') 


xlim([-.4 3.4]); 


ylim([-3.5 3]); 


3 


2 3 
DFT X, 
—e Phase Xk 
2 2:5 3 
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%* Plot of the real part of xX, 
realpart=real (Xk); 
stem(0:N-1,realpart) ; 
xlim([-.4 3.4]); 

ylim([-1.5 6.5]); 

legend ('real Xk') 


6; —® real Xk/> 


% Plot of the imaginary part of X, 
impart=imag (Xk) ; 
stem(0:N-1,impart) ; 

xlim([-.4 3.4]); 

ylim([-1.5 1.5]); 


legend ('imag Xk') 


1.5 1 1 1 
—® imag Xk 


-1/ | 


2 — 
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5.9.25 Create the function in MATLAB which computes the DFT of a sequence, x[n] 
Apply this code to the sequence x[n] = [1, —2, 2, 1], 0 <n <3. 


SOLUTION: We will present two different methods for implementing this function. 
The first way is based on a double for-loop, and the second and more complex way 
is more efficient since it uses only one loop. 


function Xk = dft(x); 
N=length (x) ; 

for k=0:N-1 

for n=0:N-1 

X (n+1) =x (n+1) *exp (-j*2*pi*k*n/N) ; 
end 

Xk (k+1) =sum(X) ; 


end 


First suggested solution: 
We run the function in MATLAB. 


x=[1 =2 2 1]; 


Xk=dE£t (x) 


The DFT of x[n] is: X[k] = {2, -1 + 31, 4, -1 — 33} 


Second suggested solution: The ‘if’ in the function ensures that x is a column 
vector. So, the multiplication in the loop is, in fact, an inner product, resulting in a 
one-dimensional element. 


function Xk =dft2 (x); 


Xk=zeros(N,1); 

n=0:N-1; 

for k=0:N-1 

Xk (k+1) =exp (-j*2*pi*k*n/N) *x; 


end 


Frequency Domain Analysis 271 


We run the function in MATLAB. 


x=[1 -2 2 1]; 


Xk=dft2 (x) 
The DFT of x[n] is: X[k] = {2, -1 + 3i, 4, - 1 — 3} 


5.9.26 Let the analog signal x,(#) = e710!!!, Plot the DFT of the signal obtained after 
sampling the analog signal with sampling frequency F, = 1000 samples/sec. 


SOLUTION: Using the following code, we plot the analog signal and the Continuous 
Time Fourier Transform. 


sAnalog signal 

Dt=0.00005; 
=-0.005:Dt:0.005; 

xa=exp (-1000*abs(t)); 
% Continues Time Fourier Transform 

Wmax=2*pi*2000; 

K=500;k=0:1:K; 

W=k*Wmax/K; 

Xa=xa*exp(-j*t’*W) * Dt; 

Xa=real (Xa) ; 

W=[-fliplr(W) ,W(2:501)];%Frequency from -Wmax to Wmax 

Xa=[fliplr (Xa) ,Xa(2:501)]; 

subplot (1,1,1) 

subplot (2,1,1);plot (t*1000,xa) ; 

xlabel('t in msec.') ;ylabel('xa(t)') 

title('Analog Signal') 

subplot (2,1,2) ;plot (W/ (2*pi*1000) ,Xa*1000) ; 


xlabel ('Frequency in KHz') ;ylabel('Xa(jW)*1000') 


title(' Continues Time Fourier Transform ') 
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Analog Signal 


tin msec. 
Continues Time Fourier Transform 


0 1 1 1 
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 


Frequency in KHz 


The following code is used to plot the discrete signal and its DFT. 


sAnalog Signal 

Dt=0.00005; 
=-0.005:Dt:0.005; 

xa=exp (-1000*abs(t)) ; 
sDiscrete Signal 

Ts=0.001;n==-5:1:5; 

x=exp (-1000*abs(n*Ts) ) ; 
SDFT 

K=500;k=0:1:K; 

w=pitk/K; 

X=x*exp (-j*n’ *w) ; 

X=real (X) ; 

w=[-fliplr(w) ,w(2:K+1)]; 

X=[fliplr(X),X(2:K+1)]; 

subplot (1,1,1) 

subplot (2,1,1) ;plot (t*1000,xa) ; 


xlabel('t in msec.') ;ylabel('xa(t) ') 
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title('Discrete signal');hold on 
stem(n*Ts*1000,x); 

gtext ('Ts=1 msec') ; 

hold off 

subplot (2,1,2);plot (w/pi,X) ; 
xlabel('Frequency in KHz') ;ylabel ('X(w) ') 
title ('DFT') 


Discrete signal 
T T 


tin msec. 
DFT 


0 1 1 if 1 1 


1 
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 
Frequency in KHz 


5.9.27 Compute the IDFT of the sequence X;, = {6,-1—j,0,-1+j},,0<k<3. 


SOLUTION: We compute the IDFT of X, according to the relation: 


.2ank 


N-1 
x)= EY Xe, 1=0,LNAL 
k=0 


Xk=[6, -1-j,0,-1+3]; 
N=length (Xk) ; 

for n=0:N-1 

for k=0:N-1 


xn (k+1) =Xk (k+1) *exp (j*2*pi*n*k/N) ; 
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end 
x (n+1)=sum(xn) ; 
end 


x= (1/N) *x 

The result is: x[n] = [1, 2,2, 1],0<n <3. 
5.9.28 Write the function in MATLAB that computes the IDFT of a sequence, X,. 
SOLUTION: As in problem 5.9.21, we will develop two functions, one implemented 


by using a double for-loop and one implemented by using an inner vector product. 
We will apply them to X, = {6, -1 —j,0,-1+j}0<k<3. 


First function for the computation of the IDFT of a sequence, X;. 


function x=idft (Xk) 

N=length (Xk) ; 

for n=0:N-1 
for k=0:N-1 


n(k+1) =Xk (k+1) *exp (j*2*pi*n*k/N) ; 


end 

x (n+1)=sum (xn) ; 
end 
x= (1/N) *x; 


We run the function in MATLAB as follows: 
SkelGe slay ,0;<149) 2 
x=idft (Xk) 


The IDFT of X;, is: x[n] = {1, 2, 2, 1}. 

Second function for the computation of the IDFT of a sequence, X,. The if of 
the function ensures that the X, is a column vector. The dot before the apostrophe 
ensures that we do not get the conjugate complex of X, Then, the multiplication in 
the loop is an inner product that produces a one-dimensional element. 


function x=idft2 (Xk) ; 
[N,M] =size (Xk) ; 
if M~=1 


Xk=Xk.'; 
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end 
x=zeros(N,1); 


k=0:N-1; 


for n=0:N-1 
x (n+1) =exp (j*2*pi*k*n/N) *Xk; 
end 


x= (1/N) *x; 


We run the function in MATLAB as follows: 
Ake[6, =1=7,/0,<1491; 
x=id£t2 (Xk) 


The IDFT of X; is: x[n] = {1, 2, 2, 1}. That is the same result as the one obtained by 
the first function. 


5.9.29 Find the circular shift, x,,[n] = x[(n — 2)mod N], of the sequence x[n] = [0.1, 0.2, 
0.3, 0.4, 0.5, 0.6, 0.7, 0.8],0 <n <7. 


SOLUTION: First suggested way: 


% We generate x[n], N and M = 2 


sWe compute (n — m)mod N for the first element (n=0). 
n=0; 

p=mod ( (n-m) ,N) 

6 The 1%* term of the circularly shifted sequence x, ,[n], 
i.e., xX, ,(0) is the 7 term of the sequence x[n], i.e., 
x[6] 

xc (n+1)=x(p+1) ; 

xcm=XC’ 

% The 2° term of the circularly shifted sequence x, ,[n], 


i.e., xX, ,(1) is the 8 term of the sequence x[n], i.e.,xI[7] 
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n=1; 

p=mod ( (n-m) ,N) 

xc (n+1)=x(p+1); 

xcm=xc’ 

% The 3**term of the circularly shifted sequence x, ,[nl, 
i.e., xX, ,(2) is the 1%* term of the sequence x[n], i.e., x[0] 
n=2; 

p=mod ( (n-m) ,N) 

xc (n+1)=x(p+1) ; 

xcm=xc’ 

% The 4** term of the circularly shifted sequence x, ,[nl, 
i.e., x, ,(3) is the 2"? term of the sequence x[n], i.e., x[1] 
n=3; 

p=mod ( (n-m) ,N) 

xc (n+1)=x(p+1); 

xcm=xc ! 

% The 5° term of the circularly shifted sequence x, ,[nl, 
i.e., x, ,(4) is the 3** term of the sequence x[n], i.e., x[2] 
n=4; 

p=mod ( (n-m) ,N) 

xc (n+1)=x(p+1); 

xcm=xc ! 

% By calculating the remaining 3 terms of x, ,[n] in the 
same way, we obtain the final value of the circularly 
shifted sequence. 

for n=5:N-1 

p=mod ((n-m) ,N) ; 

xc (n+1)=x(p+l1) ; 

end 


xcm=xc! 


The circular shift, x,,[n] = x[(1 — 2)mod N], of the sequence, x[n], is x, .[n]={0.7, 0.8, 


0.1, 0.2, 0.3, 0.4, 0.5, 0.6}. 
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Second suggested way: 


A simple way to apply a circular shift to a sequence is by using the circshift 
function. The function is compiled as xcm = circshift (x‘, m), ie., the necessary con- 
dition for the function to run is that the input vector is a column vector. The m 
expresses the shift. 


m=2; 
xcm=circshift (x',m) 


5.9.30 Let the sequences x,[n] = [1, 0, 2.5, 1.5],0 <n <3 and x,[n] = [1, 2,0.5,2]|,0<n< 
3. Use MATLAB to prove the property: DFT{x,[n] ® x,[n]} = X,[k] - X.[k]. 


SOLUTION: We define the signal y[n] = x,[n] ® x,[n] and compute its DFT (the left 
part of DFT{x,[n] ® x,[n]} = X,[k] - X,[k]). 
y = [3.75 6.75 6 6]; 
Ar=dft (y) ; 


Ar.' 


The following result is obtained: Ar = {22.5, —2.25 —0.75i, —3, —2.25 + 0.751} 

We define x,(n), x,(n) and compute their DFT, X,[k] and X,[k]. Next, we multiply 
X,[k] and X,[k] computing in this way the right part of: DFT{x,[n] ® x.[n]} = X,[A] - 
X[k]. 

x1l=[1 0 2.5 1.5]; 

x2= [0 1.52) 3 

X1=d£t (x1) ; 

X2=dft (x2) ; 

De=X1 .*X2; 


De.’ 


The following result is obtained: De = {22.5, —2.25 — 0.751, —3, —2.25 + 0.751} 


Since the two results are the same, the property of the DFT of the circular con- 
volution has been proved. 


5.9.31 Let the sequences x,[n] = [1, 2, 3, 4],0 <n <3 and xn] = [3, 2,5, 1,0 <n <3. 
Compute: 
a. The circular convolution of N = 4 points. 


The circular convolution of M = 2N-1=7 points of the sequences x,,[n] = [1, 2, 
3, 4, 0, 0, O],0 <n <6 and x,,[n] = [3, 2, 5, 1, 0, 0, 0], 0 <n < 6, which are filled in 
with zeros. 


c. The linear convolution of the sequences x,[n] and x,[n]. 
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SOLUTION: 
a. The MATLAB code for the computation of the circular convolution of N=4 
points is: 


° 


% We define x,[n], x,[n] and N 

x1=[1,2,3,4]; 

x2= [3.2 5, 1) 3 

N=length (x1) ; 

% We compute the sequence which is the circular reflection 
of x,[n] 

for m=0:N-1 

p(m+1)=mod (-m,N) ; 

end 

% We compute the circular convolution of 4 points of x, [n] 
and x, [n] 

for m=0:N-1 

X2s (1+m) =x2 (1+p(m+1)); 

end 

for n=0:N-1 

x2sn=circshift (x2s’,n); 

yl (n+1) =x1*x2sn; 


end 


28 31 22 29 


b. The MATLAB code for the computation of the circular convolution of M=2N- 
1=7 points of the sequences filled in with zeros is: 
% Definition of x,,[n] , x,,[n] and of the new N. 

x1l1=[ x1 0 0 0]; 

x22=[x2 0 0 0]; 

N=length (x11) ; 

% We compute the sequence which is the circular reflection 

of x,,[n] 


for m=0:N-1 
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p (m+1) =mod(-m,N) ; 
end 


for m=0:N-1 


X22s (1+m) =x22 (1+p(m+1)); 
end 

% We compute the circular convolution of 7 points of x,,[n] 
and x,,[n] 

for n=0:N-1 

xX22sn=circshift (x22s',n); 


y2 (n+1) =x11*x22sn; 


end 


3 8 18 29 25 23 4 


c. The MATLAB code for the computation of the linear convolution of the 
sequences x,[n] and x,[n] is: 


% We compute the linear convolution of x,[n] and x,[n] and 
% we note that is equal to the circular convolution of the 
of the sequences filled in with zeros 

y3=conv (x1,x2) 

y3 = 

3.8 18 29 25 23 4 


5.9.32 Let the sequence x[n] = 0.7",0 <n < 19. Plot both the DTFT of x[n] for0 < w < 


21k 


2x and the DFT of x[n] for m= a k=0,1, N-—1inin the same figure. 


SOLUTION: The MATLAB code is: 


n=0::19; 


syms w 
Xdtft=sum(x.*exp(-j*w*n) ) ; 
Xdft=dft (x); 

N=length (Xdft) ; 

k=0:N-1; 
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wk=2*pitk/N; 
ezplot(abs(Xdtft), [0 2*pi]); 
hold on 

plot (wk,abs (Xdft),'o') 
legend('Xdtft', 'Xdft') 


5.9.33 Let the discrete signal x[n]= cos me Find the DTFT of the signal via its DFT. 


SOLUTION: It is X(k)= X(e)|, ey: 
We plot the DTFT of the signal using the following MATLAB code: 
n=0:15; 

N=length(n) ; 

x=cos (2*pi*n*7/16) ; 

xx=fft (x) ; 
=-(N/2) : (N/2)-1; 

NN=512; 

xe=fft (x,NN) ; 

kk=- (NN/2) : (NN/2) -1; 

plot (2*pi*kk/NN, abs (xe) ) ; 


hold on; axis tight; grid on; 


stem(2*pi*k/N,abs(xx),!'or'); 
xlabel('Normalized angular frequency') ; 


ylabel ('Magnitude') ; 
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oO 


Magnitude 
AK 


-3 -2 -1 0 1 2 3 
Normalized angular frequency 


5.9.34 a. Create a function in MATLAB that computes the circular convolution of 
two sequences (which do not necessarily have the same number of samples). 


b. Create a function in MATLAB that computes the circular convolution of two 
sequences by using the functions fft and ifft. 

c. Compute, using the result of exercise (a), the circular convolution of the 
sequences x,[n] = [1, 2, 3, 4] and x,[n] = [5, 4, 3, 2, 1]. 


SOLUTION: 
a. The function in MATLAB that computes the circular convolution of two sequences: 


function y=circonv2 (x,h) 


N1=length (x) ; 
N2=length (h) ; 
N=max(N1,N2) ; 
if Nslength(x), 
x=[x zeros(1,N-length(x))]; 
end 
if Nslength(h), 
h=[h zeros(1,N-length(h) )]; 
end 
for m=0:N-1 


p (m+1) =mod(-m,N) ; 
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hs (1+m) =h(1+p (m+1) ) ; 

end 

for n=0:N-1 
hsn=circshift (hs’,n) ; 
y (n+1) =x*hsn; 


end 


b. The function in MATLAB that computes the circular convolution of two 
sequences by using the functions fft and ifft: 


function y=circonv3 (x,h) ; 
Ni=length (x) ; 
N2=length (h) ; 
N=max(N1,N2) ; 


X=fft(x,N) ; 


H=fft (h,N) ; 
y=ifft (X.*H) ; 


c. Computation of the circular convolution of the sequences x,[n] = [1, 2, 3, 4] and 
x,[n] = [5, 4, 3, 2, 1] 


x1=[1,2,3,4]; 
x2=(5,4,3:,2,411] + 
y=circonv2 (x1,x2) 
y= 
25 25 30 40 30 
y=circonv3 (x1, x2) 
y= 
25 25 30 40 30 
5.9.35 Compute and plot the FT X() of the continuous-time signal x(f) = t- 1,0 < 


t < 2. Then, compute and plot at the same figure the approximation X(Q,) for 
N=128 and T=0.1. 


SOLUTION: Description of the procedure: 
e We define the sampled signal x[nT], n = 0, 1,.... N — 1, for N=128 and T = 0.1. 
e We compute the DFT X, of x[nT] 
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e We compute the frequencies, Q,, and the approximation, X(Q,), according to 
27k 


. 1 — e N 
the relation X(Q,) = NT ——— 
j2mk 


e We express x(t) as a symbolic function and compute its FT X(Q) 


X,,k=0,1,.....N-1. 


We plot X@) and X@,) at the frequency range Q) < Q < Qy_). 
T=0.1; 

N=128; 

t=0:1/ (N*T) :2; 

x=t-1; 

Xk=fft (x,N); 

k=0:N-1; 

wk=2*pi*k/ (N*T) ; 

Xwk= (N*T* (1-exp (-j*2*pi*k/N))./ (j*2*pi*k) ) .*Xk; 
syms t w 

x=(t-1) * (heaviside(t) -heaviside(t-2)); 
X=fourier (x,w) ; 

ezplot (abs (X), [0 wk(N)]); 

hold on 

plot (wk, abs (Xwk) ,'0') 

hold off 
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5.9.36 a. Create a function in MATLAB that computes the linear convolution of two 
sequences, using the functions fft and ifft. 


b. Compute, using the function of exercise (a), the linear convolution of the 
sequences x,[n] = [1, 2,3, 4]0 <n <3 and x,[n] = [7 6,5, 4,3,2,1]|, O0<n <6. 


c. Verify the result using the command conv. 


SOLUTION: 
a. We create the function: 


function y=linconv(x,h) ; 
Nl=length (x) ; 
N2=length(h) ; 
N= N1 + N2-1 ; 


X=fft(x,N) ; 


H=fft (h,N) ; 
y=ifft (X.*H) ; 


b. We compute the linear convolution of the sequences: 
xLe1L ode 
xX2Z2=7 22121; 
y=linconv (x1,x2) 
y=7.0000 20.0000 38.0000 60.0000 50.0000 40.0000 30.0000 
20.0000 11.0000 4.0000 


c. We compute, using the command conv, the linear convolution of the sequences: 
y=conv (x1, x2) 
y = 
7 20 38 60 
50 40 30 20 
11 4 
5.9.37 Let the sequence x(n) = cos(0.482n) + cos(0.522n). Compute the 10-point DFT of 
x(n) and plot its amplitude with respect to the frequency. 
SOLUTION: 
n = [0:1:99]; x = cos(0.48*pi*n) +cos(0.52*pi*n) ; 
nl = [0:1:9]; yl = x(1:1:10); 
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subplot (2, 1, 1); stem(nl1, yl); 
title('signal x(n), 0 <= n <= 9'); 
xlabel('n'); 

Yl = dft(yl, 10); magY1l = abs(¥1(1:1:6)); 
kl = 0:1:5; wl = 2*pi/10*k1; 

subplot (2, 1, 2); plot(wl/pi, magY1) ; 
title('Samples of DFT Magnitude') ; 


xlabel('Frequency in pi units’); 


signal x(n), 0 <=n <=9 


2 T T 
0 © ° ° | © ® 
) L | 1 L Ll 1 L 
0 1 2 3 4 5 6 7 8 9 
n 
4% Samples of DFT Magnitude 
5 ee 2 
0 1 1 1 1 
0 0.2 0.4 0.6 0.8 1 


frequence in 7 units 


NOTE: The function dft.m that was used is given next: 


function [Xk] = dft(xn, N) 


% Computes Discrete Fourier Transform 


% [Xk] = dft(xn, N) 


oe 
a 
oS 
Il 


DFT coeff. Array over 0 <= k <= N-1 


oe 
% 
1] 
lI 


N-point finite-duration sequence 


% N = Length of DFT 


= [0:1:N-1]; % row vector for n 
= [0:1:N-1]; % row vector for k 
WN = exp(-j*2*pi/N) ; % Wn factor 
nk = n’*k; % creates a NxN matrix of nk values 


WNnk = WN .* nk; & DFT matrix 
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Xk 


2 
oO 


fu 


2 
oO 


oe 


oe 


= xn * WNnk; % row 
End of function 


nection [xn] = 


Computes Inverse Discrete Fourier Transform 


[xn] = idft (Xk, N) 
xn = N-point sequence 
Xk = DFT coeff. Array 
N = Length of DFT 


n = [0:1:N-1]; % row 
k = [0:1:N-1]; % row 
WN = exp(-j*2*pi/N) ; 
nk = n’*k; % creates a 
WNnk = WN .* (-nk); 


= (Xk*WNnk)/N; % row 


End of function 


idft (Xk, 


vector for DFT coefficients 


N) 


over 0 <= n <= N-1 


over 0 <= n <= N-1 


vector for n 

vector for k 

% Wn factor 

NxN matrix of nk values 
% IDFT matrix 


vector of IDFT values 


6 


Design of Digital Filters 


6.1 Introduction 


A filter is the system that processes a signal in order to detect and extract desired compo- 
nents from unwanted parts, like noise or other signals that distort it. There are two main 
types of filters, the analog and digital filters. They are completely different both in their 
physical structure and in the way they work. 


e An analog filter uses electronic circuits produce filtering effect. These filter cir- 
cuits are widely applied in video signal amplification, graphic equalizer in hi-fi 
systems, noise reduction, etc. 


e A digital filter uses a digital processor to perform numerical calculations on sam- 
pled signal values. The processor may be a PC, or a Digital Signal Processor (DSP). 


In Figure 6.1, the process of filtering a signal using digital filters is illustrated. 

The analog input signal must first be sampled and digitized using an analog-to-digital 
(ADC) converter. The resulting binary numbers, which represent successive values of the 
input signal obtained by the sampling procedure, are transferred to the processor, which, 
in turn, performs the appropriate numerical operations. These calculations typically con- 
tain multiplications of input values with certain constant numbers and summation of the 
products. In cases where this is necessary, the results of these calculations, which repre- 
sent sampled values of the filtered signal, are driven to a digital-to-analog converter (DAC) 
in order to return the signal back into the analog form. 

The most important advantages of digital filters with respect to analog are the following: 


1. A digital filter is programmable. Its operation is determined by a program in the 
processor memory, which can be easily changed, whereas an analog filter can only 
be changed by redesigning the filter circuit. 

2. A digital filter can be easily designed and implemented on a general purpose 
computer. 

3. The characteristics of the circuit designs of an analog filter (especially those con- 
taining active components) are voltage-dependent and temperature-based. Digital 
filters are not affected by such problems and are therefore particularly stable in 
terms of time and temperature. 

4. Unlike analog filters, digital filters can handle low-frequency signals with pre- 
cision. As the DSP technology continues to develop, digital filters get to be 
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Analog to Digital Digital to Analog 
converter Digital Filter converter 


Input Output 


A/D D/A 


FIGURE 6.1 
Signal filtering process using digital filters. 


implemented even to high frequency signals in the radio frequency (RF) domain, 
which was formerly an exclusive area of the analog technology. 


5. Digital filters are much more adaptable in their ability to produce signals in a vari- 
ety of ways: this includes the ability of some types of digital filters to adjust easily 
to changes of the input signal characteristics. 


6. Complex combinations of filters (for example in parallel or in series) can easily be 
implemented by fast DSP processors, making the hardware requirements rela- 
tively simple compared to the corresponding analog circuit design. 


6.2 Types of Digital Filters 


Linear digital filters are divided into two categories according to the length of their 
impulse response: The Infinite Impulse Response (IIR) filters and the Finite Impulse Response 
(FIR) filters. 

Linear digital FIR filters have a finite response to the unit impulse function input with 
the great advantage of ensuring stability and an absolutely linear phase (as long as the 
impulse response is symmetrical or anti-symmetrical to its origin)—properties that eas- 
ily explain their wide application in telecommunication systems. One disadvantage can 
be the need for a large filter length (therefore, increased computational power and large 
group delay—equal to half the length of the filter) for a sufficient approximation of the 
amplitude response. 

Linear digital IIR filters have an infinite time response to the unit impulse function 
input. For the calculation of the output, we usually use input as well as output values of 
previous instances, so we refer to them as recursive filters. These filters have less compu- 
tational complexity than the corresponding FIR filters, but they cannot ensure stability or 
linear phase, resulting in distortions. Digital IIR filters are widely applied in Digital Signal 
Processing, since all types of digital filter can be implemented as IIR filters. 


6.3 Digital Filter Design Specifications 


A digital filter is primarily used by its behavior with respect to frequency. This behavior is 
described by the frequency response of the filter both in relation to its effect on the range 
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of the harmonic amplitudes of the signal (amplitude response) and in the way it affects the 

harmonic phases (phase response). Since the frequency response of digital systems is peri- 

odic, it is sufficient to determine the behavior of the digital filter in the range from 0 to z. 
The steps for designing a digital filter are: 


a. Determine the properties required for the filter 
b. Optimally approximate the desired system by a casual time-invariant discrete system 


c. Implement the system of step (b). 
In Figure 6.2, the four basic types of filters are illustrated: 


a. Low-pass filter 

b. High-pass filter 
c. Band-pass filter 
d. Band-stop filter 


The desired magnitude and phase response of the digital filter constitute the required 
specifications for the filter design. In order to understand the specifications of a digital 
filter, let us consider the case of Figure 6.3, where the frequency response of the filter is 
illustrated (dashed line). 

In Figure 6.3, three bands are present: 


a. Pass band: The area [0,,] where the magnitude lH(e ) is within 1+6,, where +6, 
the deviation of the pass band (the required tolerance of the pass band). 


1-5, <|H(e”)<14+6, ,Vo<o, 61) 


2Tt 


FIGURE 6.2 
Filter types. 


290 Digital and Statistical Signal Processing 


|H(e”) | 


Transition 
a 


Pass band 
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| 
: Stop band 
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FIGURE 6.3 
Frequency response of an analog low-pass filter. 


b. Transition band: The area between w, and w, i.e., the pass band and stop band cut- 
off frequencies, where 6, is the deviation of the stop band (ripple) and there are no 
constraints on the response magnitude. 


6, <|H(e)|<1-6, 6.2) 


c. Stop band: The area that the magnitude (ec ) is almost zero with a deviation 6,, Le, 


H(e)/<5, , a, <lal<a (6.3) 


w, and w, are defined as fractions of n, whereas the deviations 6, and 6, are in deci- 
bels (dB). 


6.4 Design of Digital IIR Filters 


In this section, we study IIR filters on the condition that they are realizable and stable. 
Therefore, the discrete impulse response must be: 


h[n]=0 forn<0O and S ited < oo (6.4) 


n=0 
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The general form of the transfer function of an IIR filter is: 


2s Oe 
H(z)= Sani” = i (6.5) 
n=O 1+ Yaz 
k=1 


The filter of the Equation 6.5 has M zeros and N poles, which must be within the unit 
circle to satisfy the stability condition. Usually, it is M < N, and the filter is known as being 
of N order. 

The design methods of IIR filters are classified into the following categories: 


a. Indirect design methods, whereby an analog filter is, at first, designed as a standard 
(Butterworth low-pass, normalized Chebychev low-pass, elliptical low-pass, Bessel 
low-pass), where its transfer function, H(s), is determined, and it is of the form: 


M M 
0,5". (s + Zn) 
H(s) _ mo = me (6.6) 
b,,s [Is + Pr) 
k=0 k=1 
with differential Equation 6.7: 
N M 
dy(t) d"x(t) 
b =) GQ, 6.7 
2, ‘tk 2, dt™ GY) 


The indirect design methods are: 

The impulse invariant method 

The step invariant method 

The backward and forward difference methods 
The bilinear or Tustin method 


a Fw nN Pp 


The matched pole-zero method 


b. Direct design methods, whereby the digital filter is directly designed without having 
an analog standard filter designed, at first. The direct methods are: 


1. The method of He”) design 
2. The method of h[n] design 


c. Optimization methods, whereby the coefficients of the filters are calculated using 
optimization criteria. The optimization methods are: 


1. The least squares method of frequency response 
2. The linear programming method 
3. The use of an all-pass filter 


292 Digital and Statistical Signal Processing 


6.5 Indirect Methods of IIR Filter Design 


The most important points of comparison of the indirect methods of filter design to be 
studied next are: 


e Ease of use 
¢ Conservation of stability 
¢ Conservation of impulse response 


¢ Conservation of harmonic (frequency) response 


6.5.1 The Impulse Invariant Method 


In this method, the impulse response of the digital filter is obtained by the impulse 
response sampling of the analog filter. This method has the advantage of retaining the 
form of the impulse response of the analog filter to the corresponding digital filter and is 
usually applied to design low-pass and band-pass filters. 


Y(s) 


X(s) 
=Z [H (s) |. The equivalent discrete-time filter of the impulse 


For a system in which the transfer function is H(s) = 
Y@) 
X(z) 
invariant method is that whose impulse response is identical to the impulse response 
of the continuous time filter at kT time points, k = 0,1,2,3 ..., T, where T is the sampling 
period. 

The impulse response in the z-plane is the inverse z-transform of the transfer function 
H(z), whereas in the s-plane, the impulse response is the inverse Laplace transform of the 
transfer function H(s). The digital transfer function is obtained from the analog transfer 
function as follows: 


, its impulse response in the 


z-plane will be: H(z) = 


e Applying an inverse z-transform to H(s) yields h(f) in the time domain 
e By sampling h(t), h[kT] is obtained where T is the sampling period. 
e z-transforming h[kT] results in H(z) in the z-plane. 


Thus, 
H(s)—> h(t) 25 nf kT] 5 H(z) (6.8) 
or 
H(z)=Z[A[KT]], where h{kT]=[L*H(s)], 5 (6.9) 


Since the z-transform always correlates a stable pole of the s-plane to a stable pole in the 
z-plane, we conclude that the digital filter will be stable as long as the original analog 
filter is also stable. This method results in non-conservation of the frequency response 
(frequency distortion is observed due to overlapping). The step response is also not 
conserved. 
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6.5.2 Step Invariant Method (or z-Transform Method with Sample and Hold) 


The purpose of this method is to construct a digital filter, H(z), whose step response will 
consist of samples of the continuous system step response at time points KT, k= 0,1,2,3... 
where T is the sampling period. 

Consider the system illustrated in Figure 6.4. 

The digital transfer function H(z) is derived from the analog transfer function using 
Equation 6.10 or 6.11. 


1-e® 


s 


H(z) = Z[G,(s)-H(s) |= z| H6)| (6.10) 


H(z)=(1- zyz( HO) (6.11) 


s 


If the original analog filter H(s) is stable, then the equivalent discrete H(z), resulting from 
the step invariant method, will be stable. This method maintains neither the frequency 
response (harmonic response) nor the impulse response. 

Here, the zero-order hold filter (ZOH) has been used with transfer function G,(s). A DAC 
is a sample and hold circuit whose output is a partly continuous function. In Figure 6.5, the 


x(t) ; x{kT] ae WD 1 > Y(z) 


G(z) 


FIGURE 6.4 
System of sampled data using a ZOH (zero-order hold). 


f*(t) i, 


fo) f*(t) f(t) 


J (kT)d(t— kT) 


L 


>t ~t >t 


FIGURE 6.5 
The operation of zero-order hold (ZOH). 
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operation of the zero-order hold filter (with the objective to hold the last sampled value of 
the f(f) signal) is illustrated. 

Consider the system h,(f), with the impulse response of Figure 6.6. The reconstructed 
signal is obtained by convolving h,(f) with ¢*(), where g*(f) is the signal after sampling (i.e., 
gf) = g*()*h, (6), and is illustrated in Figure 6.7. 

Let f,(6), the impulse response of the hold filter, be a gate function (rectangle function), 
ie, f,(E) = u() — u(t - T) or 


F,(s)=L{u(t)—u(t-T)} = L{u()}-L{u(t-T)} 


~e 6.12 
Spice ele) 
s 8s 


Obviously, since the input is the impulse function, the transfer function of the hold cir- 
cuit is given by the following relation: 


G,(s) = (6.13) 


FIGURE 6.6 
The impulse response /1)(2). 


FIGURE 6.7 
Reconstruction of the signal with ZOH. 
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6.5.3 Backward Difference Method 


One way of estimating the derivative h(t) at t = kT is to calculate the difference of the cur- 
rent sample from the previous divided by the sampling period, as shown in Equation 6.14: 


qa HO _ MH es 


The derivative of h(t) in the s- plane is sH(s), whereas in the z-plane, it is: 


-| ne |- _| MAI Mik 11). = -H(2) 


Comparing the two derivatives, it can be deduced that the conversion of n in the z-plane 
is given by 


H(z)=H (8), ae (6.15) 


= gi 


T and z= represent the mapping of the s-plane to the 
= 


z-plane, as illustrated in Figure 6.8. By substituting s = jo, in z= ts , we obtain: 
—§ 


Relations s= u 


1 1 1+ jo,T 
1-sT|, jp, 1-j0.T  14(o,T) 


J{z} 
J{s} = jo, ae 
s-plane 
R{z} 
R{s}=o l 
S=J@ 


FIGURE 6.8 
Mapping of the s-plane to the z-plane with the backward difference method. 
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ey | 1 }-+( oT )- 1+(@,T) => 
147) “U+@T¥) © (140.7%) 


1 
x 4 2 yt 2a ys r—x4 2-95 
4 ToT? y y 
t a iy 1 
27 x4+—--—4+y?=05] x-=] +? =— 
oxet-s+y hay = (6.16) 


1 1 
Equation 6.16 is an equation of a circle with a radius of (3 and a center of (3 


Substituting s = o + ja, in z= , the same equation is derived, and the values of o 


1-sT 


iT 
define points in the circle | 575 J. 


The backward difference method conserves the stability but not the harmonic response. 
The forward difference method is not used due to instability problems. 


6.5.4 Forward Difference Method 


Another way of obtaining the derivative h(t) at = kT is by computing the difference of the 
next sample from the present divided by the sampling period, as shown in Equation 6.17. 


i= aa) _ Hk] ~h{k] 617) 


The derivative of h(t) in the s-plane is sH(s), whereas in the z- plane, it is: 


ay \_ | e+ 1)—A[k] | _ 2-1. 
Ln |=2] T - T H(z). 


By comparing the two derivatives, it follows that the transform of H(s) to the z-plane is 
given by the Equation 6.18. 


H(z) = H(s) 


wel (6.18) 
T 


The forward difference method does not always conserve the stability or the harmonic 
response, and therefore its application is usually avoided. 
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6.5.5 Bilinear or Tustin Method 


This method is based on the integration of the differential equation and then on the 
numerical approach of the integral. With this method, the transform of the function G(s) to 
the z-plane is given by Equation 6.19. 


H(z)=H(s)|_22-1 (6.19) 


(6.20) 


It is one of the most popular methods because the stability of the analog system is con- 
served while the frequency response can also be conserved considering the non-linear 
relation between the analog and the digital frequency. 

If we substitute s = jw, in Equation 6.20 (in order to map n-axis to the z-plane), we obtain: 


2 
o,T 
1l+s— ge 1-( ; + jo.T 
Zz 2 = re = > 
a ee oT 
cen J > 1+( 5 
2 
) 
; 2 oT 
Z=x+ Jy = +j 


But 


(6.21) 


Equation 6.21, i.e., x? + y? = 1, is the equation of a circle with a radius of 1 and a center of 
0, as illustrated in Figure 6.9. 
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EoRvore aineronce Backward difference Trapezoidal Appoximation 


FIGURE 6.9 
Stability regions for different methods. 


According to Figure 6.9, the left half of the s-plane is mapped by the Tustin method 
within the unit circle in the z-plane, therefore the discrete system will always be stable. 
The disadvantage of the Tustin method is the introduction of distortion on the axis of the 
frequency w due to the non-linear relation between the frequency, @, (in the s-plane) and w 
(in the z-plane). Substituting s = jw, and z = e/” in Equation 6.20 yields: 


ie a ¢lt(it__it) 2 2isine 
21-e Ze? \e% —€ 2 2 


Os Tise T ej 7%) T anim yo 
7 Pee 200s 
oT 2, @ 
@ = 2ta “(27) d o,=—tan— 
n=] and =F tan> (6.22) 


The non-linear relation between w, and w, referred to as frequency warping is illustrated 
in Figure 6.10. When the specifications of the digital filter are known, the significant analog 
frequencies are deliberately distorted (frequency pre-warping), so that the digital filter has 
the desired specifications. The limitation in the use of the bilinear transform is that the 
of the analog filter must be, in parts, linear function of o,. 


magnitude H(o,) 


FIGURE 6.10 
The non-linear relation between o, (in s-plane) and o (in z-plane). 
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6.5.6 Matched Pole-Zero Method 


A simple but, at the same time, very efficient way to find the equivalent discrete form of 
a continuous-time transfer function is the pole-zero matching method. If we calculate the 
z-transform of the e[KT] samples of a continuous signal, e(f), then the poles of the discrete 
signal E(z) are associated to E(s) poles according to z = e*’. The pole-zero matching method 
is based on the fact that we can use z-transform and the relation z = e*’ in order to locate 
E(z) zeros. 

According to this method, we consider separately the numerator and the denominator 
of the transfer function H(s) of the analog filter and depict H(s) poles at the poles of the 
discrete-time transfer function, H(z), and H(s) zeros at H(z) zeros. An important point in 
this method is the way in which the zeros and the poles in infinity in the s-plane (s = ~) 


s 


are depicted in the z-plane. The jw-axis from w = 0 to @= ue - Nyquist frequency 
in the s-plane is depicted using z = e*’ to the unit circle in the z-plane, from z = e = 1 to 
nt 


z=e'T =-1.S0, by choosing w,, the angular frequency that satisfies the sampling theorem, 
. ) . : 
we can then consider @ = a , and not @ = ~, as the maximum possible frequency. 


We can therefore assume that the frequency response G(jw) tends to zero as w tends to 


a . Similarly, we can say that H(z) (the discrete form of H(s)) tends to zero as z tends to —1. 


Consequently, point z = —1 in the z-plane depicts the maximum possible frequency, @ = 2 
The pole-zero matching is realized as follows: F 


1. All finite poles of H(s) are depicted in the z-plane according to z = e*. 
2. All finite zeros of H(s) are depicted in the z-plane according to z = e*". 


3. (i) The zeros at infinity of H(s), that is, the zeros in s = ~ are located at z = —1 in H(z). 
Therefore, for every zero at infinity in H(s), we will have a component z + 1 in the 
numerator of H(s). Similarly, the poles at infinity of H(s), if any, are depicted in the 
z=-1 point. So, for each pole ins = ~, we have a component z + 1 in the denomina- 
tor of H(z). 


(ii) If the numerator of H(z) is of a smaller degree than the denominator, then we 
add powers of z + 1 to the numerator until the numerator and the denominator are 
of the same degree. 


4, We adjust the H(z) gain. 


For low-frequency filters, it must be: H()|,_, = H(s)|,-o 

For high-frequency filters, it must be: H@)|,__; = H(s)|,_.. 

If we calculate H(z) by the pole-zero matching method, difference equations are derived 
in which the calculation of the y[kT + T] value depends on the u[kT + T] value, ie., the 
output at the time kT + T depends on the input value at the same time. However, if a 
time delay is permissible, i.e., if the output value calculation y[kT + T] needs the input 
_ Y@) 

U(z) 
denominator. In this case, we use the modified pole-zero matching method. The modified 
pole-zero matching method results from the pole-zero matching method described, by 
excluding step 3(ii). 


value u[kT], then the numerator of H (z) should be of one degree smaller than its 
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Both methods conserve the stability of the original system because, through the depic- 
tion z = e”, all points with s < 0 are depicted at z points with z < 1, since 0 < e’ < e <1. 
Let the analog transfer function be of the form 


Sb Li ceed ge) 
(oem, Meo enc lee) 


H(s)= K, msn (6.23) 


Then, based on the pole-zero matching method, H(z) will be of the form: 


(z4+1)° “(z+z,)(z+z,)....(z+z,) 


6.24 
(z+ p,)(z+ p,).....(z+p,) ( ) 


The z; and p; are matched with the corresponding pz; and ; according to relation (6.25): 
z,=-e"%" and p,=-e7" (6.25) 


The n — m multiple zeros (z + 1)""™) that appear in H(z) represent the numerator and 
denominator polynomial degree difference. 

The constant K, (dc gain factor) is chosen so that H(s) and H(z) are equal fors = 0 andz=1 
(for systems of interest in their low-frequency behavior), i.e.,: 


H(z) for z = 1) = H(s)(for s = 0) (6.26) 


So, we can easily compute the constant K, from Relation 6.26: 


ag (1+ 2,)(1.+ 25)..-(1 + Zn) =K, Hy Uy «eee ees Hin (6.27) 
(+p,)0+ py). +p.) I... 7, 


Zz 


Alternatively, constant K, can easily be computed by 


st . (6.28) 
lim y(k) lim(z a 2 pp t@ 


6.6 Direct Methods of IIR Filter Design 
_\2 
6.6.1 Design of (ec)! Method 
This method is used when the digital filter is described by a given lH(e)| and is imple- 
mented by the relation (6.29). 


ir 1 
l(c) Seen (6.29) 
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where A,,(@) is a trigonometric polynomial of n-order. A,(w) must be first calculated to 


. (2 
approximate IH(e”) for which it is: 


He”) = HOHE), (6.30) 
6.6.2 The Method of Calculating hA[n] 
Based on 
= ; 
az! we 
29___= H(z) =) hlk]z* 6.31) 


we conclude that if we want to design a digital filter with a specific impulse response /1,[k], 
k=0,1,..., h,[k], k=0,1,...,€-1, we should approach h[k] based on the given h,|k]. 
By setting the least squares error criterion, i.e., 


(e)= S°[ halk AIK] fk] (6.32) 


k=0 


we should minimize it by calculating the parameters of the filter. 


6.7 IIR Filter Frequency Transformations 
To design different types of digital filters (low-pass, high-pass, band-pass, band-reject), the 
next steps are followed: 


1. Design a standard analog low-pass filter and then transform it, using Equations 
6.33 to 6.36, into the corresponding selective analog filter, which, in turn, is trans- 
formed into a digital filter using the methods developed in Section 6.6. 


low pass > low pass s—> us (6.33) 


n 


QO, 


low pass > high pass s—> (6.34) 


low pass > band pass s—> (6.35) 
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s(@, — @,) 


low pass — band reject s——; 
s+ 0,0, 


(6.36) 


where: w, = low cut-off frequency, 
w, = high cut-off frequency 


2. We design a low-pass analog filter as a reference, which we transform into a digital 
low-pass filter, and finally, we transform the latter, using the Equations 6.37 to 6.40, 
into the desired digital filter. 


a. low-pass — low-pass 


4 ee 
Gs 


1-az" 
sin|[(o,-,)/2T | (6.37) 
~ sin{[(@, +,)/2]T | 


where: @, = cut-off frequency of the initial filter, 
@,, = cut-off frequency of the final filter 
b. low-pass > high-pass 


cos| [ (a, +0,)/2|T | (6.38) 
cos| { (@, - o,)/2]T | 


where: w, = cut-off frequency of the low-pass filter, 
@,, = cut-off frequency of the high-pass filter 


c. low-pass > band-pass 


[2° -[2ock/(k+ 1) ]z*+(k-D/(k+1)| 
[[«- 1)/(k +1) ]z? -[2ak/(k +1) ]z44 1] 
cos|[(@, +0,)/2]T | (6.39) 
cos [[(@. - o,)/2]T | 


b=cot (52) }in| 8] 


where: w, = cut-off frequency of the low-pass filter, 


-1 


= cos(@I’) = 


@, = central frequency of the band-pass filter 


Design of Digital Filters 303 


@, = low cut-off frequency 
@,, = high cut-off frequency 
d. low-pass > band-reject 


, [2° —[2a/(1+k)]z1+d- ky/(1+k)] 
ZS 
[[@ —k)/(1+k) ]z? -[20/(+k)]z1+ 1] 
cos| [ (a, +@,)/ 2\r | 
cos| | (@, -—@,)/ 2|r | 


etn] [25% Jr in| 9 


a = cos(@,T) = (6.40) 


6.8 FIR Filters 


There are critical advantages of FIR (Finite Impulse Response) filters over IIR filters, such 
as their easy implementation due to FIR filter approximation algorithms, as well as their 
stability. Their disadvantages are that they require a higher order filter than IIR filters. For 
the implementation of steep-slope FIR filters, a large number of samples is required, result- 
ing in a delay due to the convolution process, which is computationally heavy. 

If the input signal in the FIR filter is the unit impulse sequence 6[n], then its output (ie., 
the impulse response of the filter) will be successively equal to each of the coefficients h[n]. 
Thus, for FIR filters, we have a finite impulse response. It has already been mentioned that 
the difference equation of a FIR filter is non-recursive, i.e., it is of the form: 


ylnl= }°b,x[n—K] 6.41) 
k=0 
or 
yl] = hn] + xn] = Salk adn K] (6.42) 
k=0 


The order N of the filter is characterized by the number of impulse response terms (num- 
ber of h[n] coefficients). The transfer function of an N order filter will be: 


H(z)= Salle" = h[O]+ A(z +...4h[N - Iz? (6.43) 


n=0 
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By substituting z = e’°, the frequency response H(e!”) is computed: 


N- Period 20 


e lok 
no = De 


k=0 


H(ei*) = H(z) 


N-1 
H(e) = DHE (cos@k - jsinok) (6.44) 


=0 


The magnitude H (e!”), 


")= [Sc hos +{Si(vesn0s) (6.45) 


The phase ®(@) of H(e’°) will be: 


-) hfkjsin ok 
®() = ArgH(e!) = tan 5° (6.46) 

h{k]cos ak 

k=0 
For T # 1, we will have: 

wa 20 
H(e!?")= ) h[kT]e/" time period — 6.47 
(ei*") pe period (47) 

EET 


6.9 FIR Linear Phase Filters 


A filter has a linear phase response when the phase difference ®(w) between the input and 
output for a signal of angular frequency @ is given by: 


®(@)=-mo or P(w@)=b-mo (6.48) 


where m and b constants that depend on the characteristics of the filter. When the har- 
monic components of a signal pass through a system with a linear phase response accord- 
ing to Equation 6.48, all of them are subjected to the same time delay in msec. This has as a 
result that the form of the signal is maintained, i.e., the input signal is not distorted when 
it passes through the filter. 
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FIR filters can be designed to have a linear phase. A FIR filter has the linear phase 
response property if its impulse response, h[n], exhibits some kind of symmetry. 


From Equations 6.46 and 6.48, it is derived: 


N-1 N-1 
-)hfk|sin ok -)hfklsinok 
tan’! —*° =-mo => —=° = tan(-mq@) > 
h[{k]cos @k h[{k]cos ak 
k=0 k=0 
N-1 
h{k|sinok os 
i eel Y hlklsin akcosmo — 
= cosma = 4 
h{k|cos@k = 
k=0 
N-1 N-1 
— » hl{k]cosa@k sinma =0> hl] (sin ok cos mo —cos@k sin ma) =0 => 
k=0 k=0 
N-1 
SY hlk]sin(ma — ke) = 0 (6.49) 
k=0 


Expanding the sum, we have: 


h[O0]sin m@ + h[1]sin(m@ — @) + h[2]sin(m@ — 2@) +... 
+h[N —1]sin(mo—(N -1))=0= 


h[O]sin mo + h[1]sin(m-—1)@ + h[2]sin(m—-2)@ +... 
.. + h(N —1)sin(m—N +1)o) = 0 (6.50) 


Choosing the constant m= yields: 


'Holsin( “40 }+alsin{ N30 J+. 
(6.51) 
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From Equation 6.51, we conclude that, by choosing a FIR filter with symmetric coef- 


2 
with respect to the frequency and will generate a delay of the input signal by ST sec. 


ficients* of h[k] with respect to the middle (i.e., ), the filter will have a linear phase 


*i.e., it should be true that : h[k]=h[N-1-k], 0<k<N-1 (6.52) 


In conclusion, a FIR filter with a symmetric impulse response and an odd number of 
terms has a linear phase, and consequently, it introduces a time delay in all signal harmon- 
N-1 

2 
on whether the number N of their coefficients of h[n] is even or odd and whether h[n] is 
symmetric or anti-symmetric. 


ics by samples. There are four different types of linear phase FIR filters, depending 


Type I: The impulse response h[n] is symmetric, and M is odd. 


h{n] =h[M-1-n] 


(M-3)/2 
H(@) = emenal (Me) 2 {Mod ‘Jia 
k=1 


Type II: The impulse response h[n] is symmetric, and M is even. 


h[n]=h|M-1-n] 


(M-3)/2 M 1 
H@y=eorne2 Si M x) [o(e3}] 
(@)=e >, 5 cos 5 


Type III: The impulse response /hi[n] is anti-symmetric, and M is odd. 


A{n]=—-h[M-1-n] 
A{n]=—h|M -1-n] 


(MD 74 
H(o) = e floM-0/2-#/21 9 » 1 Mok ino 


k=1 
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Type IV: The impulse response h[n] is anti-symmetric, and M is even. 
h{n]=-h|M-1-n] 
(M=1)/2 


H(o) = e-iloM-0/2-"/21 9 Dy (M = ksin(olk ~1/2)) 


k=1 


6.10 Stability of FIR Filters 


In FIR filters, the output depends only on the input values x[n — k] with k = 0,..., N-1 and 
delayed output values y[n — k] are not involved in its calculation. Thus, the impulse response 
is finite, and the transfer function H(z) of the filter given by Equation 6.43 has no denomi- 
nator; consequently, it has no poles to create instability. All of this results in filter stability, 
which means that for any finite-length input signal, the output is also of a finite length. 
The position of zeros can be defined such that rejection of specific harmonic components 
of the signal can be achieved by the filter. 


6.11 Design of FIR Filters 


The procedure for designing FIR filters includes defining filter specifications, calculat- 
ing filter coefficients by one of the available methods, determining the implementation 
structure, error analysis due to the finite length of the coefficients, and choosing between 
implementation using either software or hardware. An important step in FIR filter design 
is the calculation of the coefficients h[n] so that the filter satisfies the desired magnitude 
and phase specifications in frequency. 

The methods that are widely used are: 


e The windowing method 
¢ The optimal designing method, 
e The frequency sampling method. 


6.12 The Moving Average Filters 


In the specific case where all coefficients of the FIR filter are equal to one another and equal 
to 1/ (M + 1), where M is the filter order, the output is going to be the average of the (M + 1) 
most recent samples of the input. This is a ‘moving’ average of input samples (Moving 
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Average Filter), where the value of each point is replaced by the average of its neighboring 
point values. The moving average filters, despite their simplicity, are very effective in the 
reduction of the random noise of a signal. 

The difference equation of the filter is given by: 


y[n] = (6.53) 


1 
hnl=— (¢..), n=0,1,..N-1 (6.54) 
N 
The order N of the filter is given by: 
N=2M+1 (6.55) 
The frequency response (DTFT) is: 
,2 sin(n _on-y® 
H(@)= ae Seg (6.56) 
k=0 N sin ° 
2 
or, for odd N: 
H(@)= 7711+ 2c0s@ + 2c0820 +...+2cosMo} (6.57) 


The moving average filter uses a sliding window of odd length (e.g., 3 points, etc.), replac- 
ing the value of the center point of the window with their average value. 

For example, a filter with M=1 and N = 2M+1=3 (filter length) will have 
h[n} = [hf] h[2] hB]] = (1/3) [111] coefficients and a time response that is 
derived by the sliding counterbalance of the values of the input samples, i.e., 


yl = {yf yl2L, yl je ee ee a oe ae - | as 


illustrated in Figure 6.11. 
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x[n]= [ x[1] x[2] x[3] x[4] x[5] ...] 
Se 
es 


+ <-(€/Tx) 


yinl=[ vi2] y[3] yl4] ....] 


FIGURE 6.11 
The output of a 3-point moving average filter. 


———EE ay 
6.13 FIR Filter Design Using the Frequency Sampling Method 
For the design of FIR filters using the frequency sampling method, the following are 


required: 


a. N samples (H[k]) of the desired spectrum using DFT 
b. Applying IDFT to H[k] for calculation of the discrete impulse response h[n] of the 
FIR filter. 


Taking into account that the coefficients i[n] must be real, as well as the symmetry of h[n] 


; N=1 ' 
terms with respect to the point, to satisfy the linear phase requirement, Equations 


6.58 and 6.59 are obtained for N as an even and as an odd number, respectively. 
For N as an even number, 


xa 


hn ae wt) )|2( xe oe O<n<N-1 (6.58) 


For N as an odd number, 


N-1 


ie ,0<n<N-1 (6.59) 


wk+ 2nkn) 
N 


h{n] = wr 1)* |H(k) hes 
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where 


| wet ea 
H[k]=H\e’ % = ¥ ilnle N’, -O<k<N-1 (6.60) 


H[k] =|H[kle™™ (6.61) 


For N as an even number, the linear phase condition is 


a , Osks a 
@(k) = (6.62) 
mk(N -1) N <k<N-1 
N 2 
And, for N as an odd number, the linear phase condition is 
N= 2 osksX— 
(k) = (6.63) 
mk(N —1) N-1 2pen 4 
N 2 
Likewise, for the magnitude |H[k]], it will be: 
For N as an even number, 
IHIN-k] , isks 2-1 
: : N 
Magnitude symmetry with respect to a 
|H[k] = Pe (6.64) 
0 , k= 
2 
Zero magnitude at x. 
For N as an odd number, 
|H[k]|=|HIN-k] , 1<k< <> 
(6.65) 


Magnitude symmetry with respect to cies 


If the conditions of Equations 6.62 and 6.64 are true for N=even, and also of Equations 
6.63 and 6.65 for N=odd, then Equations 6.58 and 6.59 can be applied for the design of FIR 
filters of a linear phase with real and symmetric coefficients of the impulse response, h[n]. 


Design of Digital Filters 311 


The algorithm for designing a FIR filter using spectrum sampling is briefly presented in 
the following steps: 


1. Depending on the sampling frequency f, and the number N of the samples, the 


distance of the samples is calculated by Df = & 


2. The desired magnitude of H(k) is determined, according to the specifications for 
k = 0 to (N-1)/2+1 for an odd N, or N/2 for an even N. (The magnitude is defined 
according to the specifications and taking into account the distance between the 
samples.) 


3. Calculation of the coefficients h(i), depending on whether N is even or odd, for i = 
0,1,2..N - 1. 


6.14 FIR Filter Design Using the Window Method 


The window method is a relatively simple FIR filter design process. The ideal filter coef- 
ficients h[n] are calculated by the inverse Fourier transform of the frequency response 
He"). 

The method starts with the implementation of an ideal low-pass filter. In the following 
Figures 6.12 and 6.13, an ideal low-pass filter is illustrated firstly with respect to the digital 
frequency, w, and then with respect to the normalized frequency, F. 


|H (w) | 


FIGURE 6.12 


Desired H@)-Ideal low-pass filter @ = QT = 2aF = 27 =. 


FIGURE 6.13 
Desired H(F)-Ideal low-pass filter with respect to the normalized frequency F. 
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This method is recommended for the cut-off process of the central part of N length of the 


infinite impulse response. If W[n] is the window sequence, then it is true that: 


h{n] = h[n]W[n] 


where h[n] is the impulse response (of infinite length) of the ideal filter, ie., 


h{n|= x [rilei*)ei do 


i i “ ‘ jn@ 1 : 
=> h[n]= = H(o)e!"? dw = fT 1. da = 1 e = sin(n@,) 
20 on 2n| jn a 
a i 


-O| 


and 


Win] = #0 E -NpcnsN/ 


0 F elsewhere 


(6.66) 


(6.67) 


(6.68) 


(6.69) 


We can see that the filter described by Equation 6.68 has an infinite impulse response, 
so it is an IIR filter. We can then create an FIR filter by choosing the suitable window. In 
Table 6.1, the formulas for the calculation of h[n] coefficients for all types of filters are 


given. 


TABLE 6.1 
Ideal Impulse Responses 
Filter Type h(n) Coefficients Width 
Low pass h{k]= alanis) 2F, 
mk 
High pass h{k] =-— onan) 1-2F, 
T 
md pase hk] sin(27F,k)  sin(27E,k) 2(F,-F;) 
mk mk 
Band stop ial a AS Lo] =0in2F 4 


1k mk 
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Function W[e/*], which is the spectrum of window w[n], consists of a main lobe contain- 
ing most of the spectral energy of the window, as well as side lobes that generally have 
rapidly decreasing amplitudes. The width of the transition band depends on the width of 
the main lobe, W[e/*]. The side lobes also influence the form of the final W[e/], introduc- 
ing small ripples at all frequencies. Thus, the spectrum of a window function must have 
a narrow main lobe that contains as much spectral energy as possible, with the side lobes 
rapidly decaying. 

The main window functions are the following: 


1. Rectangular (Figure 6.14): It corresponds to the simple cut-off process of the 
sequence h,[n]. The spectrum of the window function is given by: 


W[nJ=1 , O<n<N (6.70) 


N 
2\n— > 
Wii]=1-- =, 0<n<N (6.71) 
3. Hanning (Figure 6.16): 
W[n]= 0.5 -0.5 cos( 2%") , OS<nsN (6.72) 
ies Time domain Frequency domain 
1 
0.8 
E 0.6 a 
0.4 = 
0.2 
10) 
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FIGURE 6.14 
64-point Rectangular Window. 
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Window Viewer 
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FIGURE 6.15 
64-point Bartlett Window. 
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FIGURE 6.16 
64-point Hanning Window. 


4. Hamming (Figure 6.17): The main lobe of W[e/] contains 99.96% of the total spectral 
energy, while -52 dB attenuation is achieved in the cut-off zone of the filters being 
designed. 


Wn] = 0.54— 0.46c0s{ 22") , OSn<N (6.73) 


5. Blackman (Figure 6.18): 


W(n] = 0.42 -0.5 cos( 7 +0.08 cos( 4"), O<n<N (6.74) 
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Window Viewer 
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64-point Hamming Window. 
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FIGURE 6.18 
64-point Blackman Window. 


6. Kaiser (Figure 6.19): It has most of the energy in the main lobe for a specific width 


of the side lobes. 
1 BYt=[-09/ a} 


1,[B] 


where: a = N/2 and I,(:) is a zero order modified Bessel function of the first kind, 
which can be generated by the power series: 


co k 2 
Iy(x) = 1+) 02) (6.76) 


Win] = 


O<n<N (6.75) 
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FIGURE 6.19 
200-point Kaiser Window with a beta f = 2.5. 


The parameter f (beta) defines the shape of the window and, thus, controls the changes 
between the width of the main lobe and the width of the side lobes. 

There are two empirical equations, the first of which associates the ripple of the stop 
band of a low-pass filter (a, = —20 log 5,) with the parameter, B (beta), while the second 
associates N with the transition band width of Af, and the attenuation, o,, at the stop band. 


B=0.1102(a,-8.7), a, >50 


B =0.5842(a, — 21)°* + 0.07886(a, — 21), 21<a, <50 (6.77) 
B=0, a,<21 
a, —7.95 
-“s 6.78 
14.36Af 678) 


The method of designing FIR filters with the window method, although simple, does not 
eliminate the main disadvantage of FIR filters, which require many coefficients, thereby 
increasing the hardware requirements for their implementation. However, the proper use 
of the windows can contribute to the reduction of the hardware requirements. 

In Table 6.1, the ideal impulse responses of various types of filters are given. 

The filters in Table 6.2 have an impulse response of infinite length and are, therefore, IIR. 
Table 6.2 associates the choice of the window function with the desired signal attenuation 


in the stop band. 

The choice of the length, N, depends on K and AF, where K (see Table 6.2) depends on the 
choice of window, while AF = bah is the transition band of the digital frequency. The 
following should be true: 


(6.79) 
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TABLE 6.2 


How the Choice of the Window Function Associates with the Desired 
Signal Attenuation in the Stop Band 


Width of Maximum Signal 

Transition 5, Attenuation in the 
ala Window Type Band (Hz) (db) Stop Band (dB) K 
1 RECTANGULAR 0.9/N 0.7416 21 0.9 
2 HANNING 3.1/N 0.0546 44 3.1 
3 HAMMING 3.3/N 0.0194 53 3.3 
4 BLACKMAN 5.5/N 0.0017 75 5.5 


The steps followed to design a FIR filter using the window method are: 


Ls 
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Based on the pass band specifications of the filter to be designed, a window is 
chosen, and the number of filter coefficients is calculated using the correct relation 
between the width of the filter and the width of the transition band. 


2. Calculation of the N order of the filter. 

3. Calculation of the coefficients of the ideal impulse response of the filter. 

4. Shift of the ideal h[n] to the right by (N — 1)/2 according to: 
Wisse = Vagus [n = ). n=0,1,2,.....,(N —1) (for N odd, otherwise by N/2). 

5. Calculation of the finite length sequence h[n] of the FIR filter by multiplying the 
ideal h[n] with the window h[n], 

h[n]= hy ipeal[n], OS n <(N -1) (6.80) 
eS 


6.15 Optimal Equiripple FIR Filter Design 


The term optimal FIR filter means achieving the best possible specifications of the frequency 
response with the smallest number of coefficients. The method of designing optimal FIR 
filters was developed in the early 1970s and is encoded into a computer program, making 
it easy to apply. With this method, the approximate error between the ideal and the actual 
FIR filter spreads evenly over the entire frequency range, resulting in the approximation of 
the FIR filter with a lower order filter than the one that approximates the ideal filter at one 
frequency, while at other frequencies, its approximation is not good. 

Consider a Type 1 FIR filter with respect to N, where M = N/2 and h[n] are the impulse 
response coefficients. The frequency response will be: 


M 
H(@) = ye, cos@n 


n=0 


(6.81) 
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,n=1,2,3 (6.82) 


Figures 6.20 and 6.21 illustrate the desired and the actual frequency response, respectively. 
With this method, the error is E@) = D@) — H(@) and W(a) in the pass band is equal to 


6. 
—. while in the stop band, is equal to 1. The equalized error is E,,(@) = W(w)[D(@) — H(@)] or 


1 


E.,(@) = W(@) 


Dio) - ¥°, ct (6.83) 
n=0 


To achieve equiripple, it is sufficient to calculate b,, so that the magnitude of the maxi- 
mum E,,(w) is minimized for each @ (of all frequency bands). 


|D(w) 


Normalized frequency 


(rad/s) 
SS eS Lo 
ie} Wp Ws ™ W 


FIGURE 6.20 
Desired frequency response. 


FIGURE 6.21 
Actual frequency response. 
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The method of optimal equiripple filter design, also known as Parks-McClellan, leads to 
optimal filters in the sense of the mean square error criterion from the desired response. 
The resultant filters present equiripples in both the pass band and the stop band. The 
length of the two-zone Parks-McClellan filters (e.g., low-pass or high-pass) is determined 
by the following empirical equation: 


7 - 2llogio (6,6, )—13 
7 2.324 @ 


+1, Aw=2n/(f,-f,) (6.84) 


where 6, and 6, represent the relative ripple magnitudes in the two zones. The Parks- 
McClellan algorithm keeps the quantities N, w,, @, constant in order to control the bound- 
aries of the bands and allows 6, and 6, variation. The number of coefficients required are 
known in advance. 

The steps followed to design a FIR filter using the optimal method are: 


1. Identification of the boundaries of the bands, of the ripple magnitude of pass band 
(6,) and the stop band (6,) and of the sampling frequency (f)). 


2. Normalization of the band frequencies and of the width of the pass band (Af), ie., 
division by f.. 


. Calculation of the filter size, N, by using 6,, 6, and Af. 
. Calculation of the ratio 6,/6, as the weighting coefficients. 


oF WwW 


. Use of algorithm to calculate filter coefficients. 

6. Check if ds and 6p satisfy the specifications. 

7. If no, increase N and repeat steps (5) and (6) until step (6) is satisfied. 
8. Check of fulfillment of the frequency response specifications. 


6.16 Comparison of the FIR Filter Design Methods 


A comparison of the FIR filter design methods is given next in Table 6.3. 


TABLE 6.3 
Comparison of the FIR Filter Design Methods 
Design Method Advantages Disadvantages 
WINDOW Simple to understand. It does not allow absolute 
Suitable when rippling in control of band boundaries 
the bands is the same. values. 
OPTIMAL Simple and flexible. Necessity of algorithm use. 
There is absolute control of 
the filter specifications. 
FREQUENCY Applies to recursive and Lack of frequency boundaries 
SAMPLING non-recursive filters. control. 


Filter design is accomplished 
with any magnitudes-phase 
responses. 
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6.17 Solved Problems 


6.17.1 Find the equation of a digital filter resulting from the transform of the analog 
filter with the transfer function 


H(s)= eee 
(s+1) +4 
applying the pole-zero match method for T=1sec. 


SOLUTION: H(s) can be written as: 


stl 
(s+1+2j)(s+1-2)) @) 


H(s)= 


Because the denominator degree is n = 2 and the numerator degree is m = 1, it 
is assumed to have n-m = 2-1 = 1 zero at infinity. Note that the H(s) zeros that tend 
to infinity are substituted by z =—1 in the z-plane. So, H(z) will be: 


(daze?) les") 
H(z)= 
@) 1—2ze7 cos(2T)+e7'z* 2) 


Explanation of the term explanation of the term 1—z“e-: The frequency of the 
s-plane, w, — co (when zero gets to infinity) is equivalent to the frequency of the 
z-field, so that the match of the zero at infinity with the zero in the z-plane is: 


,O 
J— zeit yz te/T! = 4-2 a1 z‘(-l)=14+2z" (3) 


(2),(3) 


aoe (l¢z")\—z'e7) 


1-2z1e cos(2T) +e" Zz" 


(z+1)(z-e") 
z? —2ze™ cos(2T) +e" 


=> H(z)= 


For T = 1sec, it is: 


_ (z+1)(z- 0.3678) 
(4) => H(z)= 2 +0.306z + 0.135 7 


but, 


Y(z) > 2? +0.6322z— 0.3678 


H(z 
(2) X(z) sz? + 0.306z + 0.135 
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Equation (6) must be multiplied by the dc gain correction factor, i.e., 


H(2)= Y@) _, 27 +0.6322z— 0.3678 
#2? 40.3062 + 0.135 


where k,, is calculated as follows: 


_ iimy( 
lim y(k) 


L.T 
lim y(f) = ims{ 1} 0) = lim H(s) =0.2 
oo sa S sa 


aor Zz ; 1.2644 
lim y(ke) = lim(z — 1) H(@) = limzH (2) = 57 = 0.877 
(8) 3 k= = 0.23 
0.877 
1 2 
Garg gg ee ee 
X(z) 140.306z7! + 0.135z 


(14 0.30627 + 0.13522) Y(z) = 0.23(1+ 0.632227 — 0.3678z2) X(z) > 
Y(z) = 0.135zY (z) — 0.306zY(z) + 0.23X(z) + 0.1454zX(z) — 0.084z2X(z) 


Therefore, the digital filter of the difference equation will be: 


y[k] = -0.135y[k — 2]-0.306y[k - 1] 
+0.23x[k] + 0.1454x[k — 1]- 0.0846x[k — 2] 


6.17.2 Convert the analog band-pass filter with the transfer function 


1 


HA(s) = ———_.—_ 
(s+0.1) +9 
to a digital IIR filter using: 

a. The Backward Difference method 

b. The Impulse Invariant method 
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SOLUTION: 
a. Using the Backward Difference method, it is: 


1 
H(z) = H(s)|_ 1-24 = = 
T J-z! (2) 
+0.1] +9 
T 
= Hite) T?/(1+0.2T +9.01T?) 
~ 204001) oa, 1 4 (3) 


Z Z 
1+0.2T +9.01T* 1+0.2T +9.01T? 


For T = 0.1sec, the poles are: P;, = 0.91 + j0.27 = 0.949e+/6> 


That is, they are located close to the unit circle (the same happens if we 
choose T<0.1). 


b. H(s) is expressed as a sum of fractions: 


H(s)= % + 4 


s+0.1-j3 s+0.14+j3 


(4) 1 1 
ths: Hey=Z{H)]92| 4 foo] fs 


s+0.1-73 s+0.1+ 73 


=> H(z) ' % 


= : ote : 
1—e OT BT © 4p OT, BTA 


1—(e" cos3T) 27 
—(2¢T cog 3T) 271 + e724 


eG 


The magnitude of the frequency response for this filter is illustrated in the 
following figure for T = 0.1sec and T = 0.5sec. 
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Magnitude (dB) 
8 6 o = 


0 0.1 0.2 0.3 0.4 0.5 


Normalized Frequency 


6.17.3 The transfer function of an analog Chebyshev, low-pass filter of order n = 4 and 
cut-off frequency @,, = 0,2a is given next. Design a digital high-pass filter with 
the cut-off frequency , = 0,62 and AT = 1sec using the Tustin method. 


7 0.038286 
(s? + 0.4233s + 0.1103)(s? + 0.1753s + 0.3894) 


H,(s) 


SOLUTION: We apply the Tustin method: 


H(z)=H,(9)|_ 21-21 > 


T4421 


0.001836(1+ z7)* 


H(z)= 1 
®) (1-1.5548z7' + 0.6493z)(1 — 1.4996z* + 0.8482z 7) @) 
To design the requested high-pass digital filter, we use Relation 6.38, so: 
cos||(@,+@,,)/2 |T 
K ) | | _ _9.3090169 _ 0.38197 (2) 


cos| | (a, -@,)/ 2\r| 0.8090169 
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The transfer function of a high-pass filter will be: 


H(z) = H(@)| 1 lta zt038i97 > 
ltaz! 1-0.38197z1 


0.02426(1— 27)" 
(1—1.0416z7 +. 0.4019z)(1—0.5561z7 + 0.764722) 


H,(z) _ 


The difference equation describing the system is derived from: 


H,(2)= Xo = Y(z) = H,(2)X(z) = yin) =z" | H,(2)X(z) | 


and is implemented using the structures already discussed, i.e., Direct form I, 
Canonic form, Cascade form or Parallel form. 


6.17.4 Let the transfer function of a low-pass Chebyshev filter of order n = 3 


0.4431 


H = 
als) (s + 0.4606)(s + 0.2303 — j0.9534)(s + 0.2303 + j0.9534) 


Find: 
a. The frequency response of H,(s) (magnitude-phase). 


b. The equivalent to the impulse invariant method digital filter (magnitude— 
phase of H(z)) 


The impulse response /1,(f) of H,(s). 
The impulse response /i[nT] of the digital filter. 
The filter difference equation 


mo an 


The realization of the digital filter, using the Parallel form, for T = 1sec. 


SOLUTION: 
a. To find the frequency response of H,(s), we substitute s = ja, which yields: 


H,(jo) = 0.4431 F 
0) ie + 0.4606)( je + 0.2303 — j0.9534)(ja + 0.2303 + j0.9534) (1) 
The magnitude |H,(ja)| is: 
H,(jo)|= = | = 
|j@ + 0.4606]|0.2303 + j(@ — 0.9534) 0.2303 + j(@ + 0.9534)| 
0.4431 
H,(jo) 2 2 2 2 2 2 a (2) 
(0.4606)? + (0.2303)? + (@ — 0.9534)? (0.2303)? + (w + 0.9534) 
0.4431 
H, (jo) 


0.2122 + 0.05303 +(@ — 0.9534)" 0.05303 +(@+0.9534) 
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The magnitude |H,(je)| in dB is: 


20 log |H, (j@)| = -7.07 — 20 log V0.2122 + w” — 20 log (0.05303 + (w — 0.9534)? 


(3) 
— 20 log 0.05303 + (w + 0.9534)? 
and Arg(H,(ja)) is: 
~ 0.9534 @+0.9534 
Are(H =-tan?—? tan ® 4 
r3(Ha(joo)) tan” 5 eng Gongs BP a593 (4) 
NOTE: Another way of the derivation of magnitude and phase of H,(jm) would be: 
0.4431 
A,(j@)= ( ar a 
0.4431 — 0.9212w") + j(1.1742m- w°) 
0.4431 
|H,(jo)| = 5 5 
(0.4431 - 0.921207)’ + (1.17420- 0”) 6) 
1.17420 -@° 


Arg(H,(ja)) =~+tan™ 


0.4431 — 0.92127 


The magnitude and phase diagram of H,(s) are illustrated in the following 


figures: 
db [| 
| H(w) | 
H(e!’AT) 
oe 
| H(w) | 
H(e!’AT) 
AT=1s 
fe) 1 2 3 


w(rad/s) 
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Phase 
deg H(e@AT) 
180 AT=1s 
v 
120 
60 t 
fe) |H(Gw)| 
60 
H(e@AT) 
=a20 ee AT=1.5s 
-180 
oO 1 2 3 
w (rad/s) 
H@s 0.4431 
* (s + 0.4606)(s + 0.2303 — j0.9534)(s + 0.2303 + j0.9534) 
0.4431 
= > 
(s-+0.4606)| (s-+ 0.2303)? + (0.9534)? | 
H,,(s) k, ks +k, 


= + 
$+0.4606  (s+0.2303)? + (0.9534) 


where k, = lim ag tt (SVS + 0.4606) = 0.4606 and 


——0.46 


k, [(s + 0.2303)* + (0.9534)? +(k,s + ks )(s + 0.4606) | =0.4431> 
k, +k, =0= k, =-k, =-0.4606 
0.4606(k, +k,)+k, =0>k, =0 


Thus: 


H, (s) = 04606 0.4606s 6 
oe" $+0.4606  (s+0.2303)? + (0.9534) 


The transfer function of the digital filter of the impulse invariant method is: 


(6) 


Hey z[H,(s) | =7 0.4606 0.4606s 


$+0.4606 (s+0.2303)" + (0.9534) 


Design of Digital Filters 327 


0.4606 0.4606 
H(z)=Z Z —* ; (7) 
5 + 0.4606 (s + 0.2303)" + (0.9534) 
0.4606 0.4606 z-0.4606 
But Z|. + se] ~ 4 lp 0400eT ~ 5 0.46067 and 
0.4606s iy 0.4606] s + 0.2303 — 0.2303 | 
(s + 0.2303) + (0.9534)? | | (s +0.2303)? + (0.9534)? 
= 0.4606Z ee | oer dita : 
(s + 0.2303) + (0.9534) (s + 0.2303)? + (0.9534) 


2° — ze T cos(0.9534T) 
= 0.4606 
2? —2ze°3T cos(0.9534T) + eo" 
70923087 gin().9534T) 
2? —2ze 9 3F cgs(0.9534T) + eT 


— 0.11128 


Thus, 


(7) 2 _ ,,-0.2303T 
=> H(z) = 0.4606 Zz Zz —ze cos(0.9534T) 
Z 


_ e 0-4606T 2 = 2 ze70.2308T cos(0.9534T) a e 0-4606T 


ze °?T sin(0.9534T) 
2? — 2ze 9 B8T cos(0.9534T) + e 04? 


+ 0.2416 


The requested frequency response of the digital filter is obtained by substi- 
tuting Z = eet 


. H( jo) = @Ae0E er gi2OT _ pjoT p-0.2303T cos(0.9534T) 
: “os eat _— e 0-4606T e/2or _ DQelOF p~0.2303T cos(0.9534T) + e0-4606T 
ete 0287 sin(0.9534T) 
j20T _ 9 pie 9-0.2303T cos(0.9534T) + 0.46067 


9) 


+ 0.2416 
e 


c. The impulse response h,(¢) of H,(s) is: 


? (6) | 0.4606 0.4606s 
hal (Hepa l 3 ; 
s+0.4606 (s+0.2303)? + (0.9534) 
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_, [| 0.4606 s +0.2303 
s+0.4606 ~~ (s+0.2303)? + (0.9534)? 
0.9534 


+0.11128 
(s +0.2303)? + (0.9534)? 


h(t) = 0.4606] e049 — e923" cgg(0,9534t) + 0.2416e°™ sin(0.9534¢) | (10) 
d. The impulse response of the digital filter is: 


(10) 
h[nT] = Z[ h,(t) | = 0.4606[ 4" — 6-25" egg(0.9534nT) ‘i 
+ 602308" sin(0,9534nT) | 


h[nT] can also be derived from the inverse z-transform of H(z) (Eq. 8), that is 
h{nT] = Z-“[H(2)]. 
e. The difference equation of the filter is derived as follows: 


H(z) = 20.4606 1 1—z1¢°25 cos(0.9534T) 
X(z) . 1- e 0-4606T 5-1 lee Iz 190.2303 cos(0.9534T) a 22,e-0-4606T 
-1 ,-0.2303T 
+0.2416 ze cos(0.9534T) 
1—2z71¢T cos(0.9534T) + ze 0407 


= H(z) = 0.4606[ H,(z)+ H,(z)+H,(z) | 


Y,(2)_, Yo(2) , Yo(2) _ Y@) 2) 
X(z) X(z) X(z)} X(z) 


= 0.8606] 


where 


Y(z) = (Y;(z)+ Yp(z) + Yq(z))- 0.4606 (13) 


We can see that the digital filter consists of three subsystems connected in 
parallel. Each of them has a difference equation: 


Y(z 1 - 
a ~ 1 — ¢0-4606T 1 >(1 erry nN) = X(z)=> 


LZAT: 
Vi@akes+err' sos 
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y,[nT] = x[nT]+ 00 *y,[(n-1)T] 


Y,(z) 1 — te 287 co5(0.9534T) 
X(z) 1-227 te 99 cos(0.9534T) + ze 040? 


Y,(z) =—X(z)+ e°?" cos(0.9534T)z 1X(z) + 2e °°" cos(0.9534T)z 'Y,(z) 
LZaT: 
= gete ty. (z)=> (15) 


y.[nT]=—x[nT]+e°*" cos(0.9534T)x[(n —1)T]+ 2e°*" cos(0.9534T)y,[(n- 1T] 
_ mas (Ca! a 27] 


=> 


Y3(z) _ 0.2416 z te 9087 sin(0.9534T) 
X(z) 1—2z71¢? 8"F cos(0.9534T) + ze OA? 
Y3(z) = 0.2416e °°" z"X(z)- sin(0.9534T) + 2e °°" cos(0.9534T)z 'Y;(z) 
P22: 
_ ee eC) => 
[nT] = 0.2416e°*" x[(n — 1)T] sin(0.9534T) + 2e°**" cos(0.9534T )y3[(1 - DT] 


= ery, [(n _ 2)T] 


(16) 


f. Relations (14), (15), (16) for T = 1sec yield: 


(14) => y,[n] = x[n] + 0.6309y,[n- 1] = 


Y,(z) 1 17) 


X(z) 1—-0.6309z7 


(15) = y,(n) = —x(n) + 0.4598x(n — 1) + 0.9196y,(n— 1) — 0.6309y,(n— 2) 
Yo(z) _ —(1—0.4598z 7) 
X(z) 1-0.9196z7' + 0.6309z7 


(18) 


(16) > y3(n) = 0.1565x(n — 1) + 0.9196y3(n — 1) — 0.6309y,(n — 2) 
Yo(Z) _ 0.15652" 
X(z)  1-0.9196z7' + 0.6309z7 


(19) 


The requested parallel realization of the digital filter is illustrated in the fol- 


lowing figures. 
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Y1[nAT] 


x[nAT] 


y[nAT] 
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y3[NAT] 


c. Filter with T.F. H3(z) 


6.17.5 Calculate the discrete transfer function, H(z), if the corresponding analog is 
oe S+ 2 1 ) 
(s+0.1)° +9 
approximately the frequency response of H(z) and compare it to the frequency 
response of the analog system. 


T =0.1sec, using the impulse invariance method. Plot 


SOLUTION: 


_ st+0.1 
oe (s+0.1)° +9 @) 


Using the impulse invariant method, it is: 


H(z) = Z(H(s)) = z| 


s+0.1 2? —e°" cos3Tz 
(s+ 0.1% 49] z?-2e' cos3Tz+e°7" 


esa. \{0.01+ w2 ay 


1 AG s}| ~~ 
Dee) (J@..0, +9] (9.01- 2)? + 0.0400? 


le? _ el cg 0,3¢| 


2) = |H(e”)|=— ; 
ave) le? — 2¢ 980,36 + e%| 
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\cos 2@ + jsin2@ — 0.946(cos @ + jsina) 


\cos 2 + jsin 2 - 1.8916(cosw + jsinw) + 9802| 


\(cos2@ — 0.946 cos)? + (sin 2m — 0.946sin a)” 


= |H(e")| = 


(0.9802 + cos2@— 1.8916 cos@)’ + (sin2@ — 1.8916sin @)” 


(2)' 


Based on (1) and (2), we plot the frequency responses of the analog and digital 
filter as illustrated in the following figures. We notice that for T = 0.1sec, the fre- 
quency response is closer to that of the analog system (in the second figure, the 
plot for T = 0.1sec and T = 0.5sec is illustrated to show the necessity of choosing a 
short sampling period). 


Magnitude (dB) 


10 


Magnitude (dB) 


6.17.6 Compute the discrete TF H(z) if the corresponding analog is H(s) = 


20 


0 2 4 6 8 10 


Frequency 


0.4 


0.3 0.5 


Normalized Frequency 


2 
(s+1)(s +2) 


with the backward difference method. Plot the frequency response of H(z) and 
compare it to the frequency response of the analog system. 
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SOLUTION: Using the backward difference method, the transfer function of the digital 


system is: 


_ 2 _ 22? 
~~ Q-27°93-27) (z-1)3z-1 


H(z) = H(s)|_1-2 = 


2el2 


ze (Qe! —1)(3e!” - 1) 


H(e!”) = H(z) 


The magnitude and phase of the digital filter are computed as follows: 


2 
(2080 - 1) +4sin®w \(3cose 7 1) +9sin*@ 


2sin @  3sin@ 
tan 


ite" 


¢(@) = 2@- tan" 


2cos@—1 3cos@-1 


For the analog system, it is true that: 


2 


HYO)= HOS). Gao jor 2) 


The magnitude and phase of the analog filter are computed as follows: 


2 


V1+@? 440? 
_1 @ 


H(jo) = 


@(@) = — tan” w — tan 


The frequency responses of the analog and the digital filters are designed in the 
following figure. Note that, with the backward difference method, a quite accurate 


reproduction of the analog system to z-plane is obtained. 
dB 
| H(jw) | 
|H(e"*) | 


|H(jw)| 
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6.17.7 Design an FIR low-pass filter with cut-off frequency , = 0.4x and linear phase. 
(Choose N=15.) 


SOLUTION: According to the specifications, the magnitude and phase of the low-pass 
filter are: 


A |H(e)| 


W,=0.4r : on 


For N=15 (odd), 


IH(k)|=|H(5-k| , 1<ks<7 (1) 
— , OSKS7 
a= Q) 
EY. Pepeia 


15 15 
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The sampling intervals of H(e’*) are +. a The following figure illustrates 


the ideal magnitude and phase specifications for N=15. 


4 | H(e")| 
k 
1 3 12 DS 
2m 0.4m Tt 2m 
otk) a 
O(k) A 15 


> 
Tt 2m wye2r : 
/ 15 


-14nk 
15 


According to Equation 6.54, the impulse response /[n] of the FIR filter is: 


= “A 1) * H1(1)|cos 2) , 0<ns14 (3) 


H(0) = H(1) = H(2)= H(3)=1 
H(4) = H(5)= H(6)= H(7)=0 


. So, Equation (3) becomes: 


h[n] = ate 1)|H(k) cos 2a) ; 


O<n<14 (4) 
15 15 


nO eee (2 tyes 3:2 eos” 4a? cos *r) 
i516 15 15 15 
h{0] = «= + at 1.95629 + 1.827091 — 1.618034) = —0.0498155 (5) 
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Likewise, 


h{1]= 0.0412022 , h[2]= 0.0666667 ,h[3] = —0.0364878, h[4] = —0.107868, 
h[5]= 0.034078, h[6]= 0.318892, h[7]= 0.466667, h[8]= 0.318892, 

h[9] = 0.0340779 , h[10] = —0.10787, h[11] = —0.036488, [12] = 0.066667, 
h[13]= 0.0412022, h[14] = -0.0498159 


6.17.8 Design the digital Butterworth filter based on the following specifications: 
20log,)|H(e”)|>-0.5 dB, 20log,)|H(e™)|<-15 dB, where w, = 0.252 and 
@, = 0.552. 
Use: 
a. the impulse invariant method, and 
b. the bilinear method. 


SOLUTION: 
a. The impulse invariant method 
We choose as a sampling period T = 1s, since it does not affect the design of 
the filter. The corresponding analog frequencies (Q = w/T) will be Q, = 0.252 


and @, = 0.552. Therefore, the given specifications for the continuous-time fil- 
ter are written as follows: 


20log 1 H, ( j,)) >-0.5 dB, 20log 19|H, ( jQ,)| <-15 dB, 


or: 


H,(jQ,)| =10°", |H,(ja,/ < 107. 
P 


The squared magnitude of the Butterworth filter of order N is: 


Now, considering only the equalities to the above equations and the form of 
|H,GQ)|?, we obtain the following system of equations with respect to N and 


c 


re) 2N O 2N 
1+{ 2] =10°% and 1( 2) =10'5, 


c 
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We rewrite them in the form 


ee Q, 2N 7 10° —1 
10!° =| 


resulting to the following value for N: 


( 10° —1 
108) Soe 
nN=—\107=1/ _ 3.5039, 


7 Q 
2og| 2 


Since N must be an integer, we choose the order of the filter by rounding the 
above result to the nearest integer, i.e. N=4. By substituting it to the first of the 


8 
two equations, the following is obtained: (2) = 10° -1 and solving for Q.: 


¢c 


Q, _ 0.252 


Q. = = = 1.0216. 
“(10° —1)'/8 ~— (0.7688 


This value satisfies the specification for the stop band. It is: 
| H,, (jQ,) P= 0.0147 < 0.0316 = 107. 


The poles of H,(s)H,(-s), which are located in the left half of the complex 
plane (leading to a stable filter) are: 


Sp = —0.3909 + 7 0.9438,s, = —0.9438 + 0.3909 j as well as their conjugates. 
Thus, the following transfer function is derived: 


H,(s) = . 
(s — 5) )(s — 5 )($ — $,)(s— $7) 
QY 
~ (8? —2Re(s,)s+ | 8 |?)(s? —2 Re(s, )s+ | s, 7) 
1.0892 


~ (5? +0.7818s + 1.0436)(s? + 1.8876 + 1.0436)" 
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Rewriting the above function as a sum of partial fractions, we obtain: 


a b c d 
A, (s) = 7 = 
S— So S—Sq S—S, S—S, 


where a = —0.4719 + j 0.1955,b = a’,c = 0.4719 — j1.1393,d = c’. The transfer function 
is then: 


a a* Cc ct 


A(z)= + + + 
1-e%" zt lee" se 


J-eiTz4 
2 Re(a) — 2 Re(a* e" )z" , 2 Re(c)-2Re(c* ez! 
1-2 Re(e%" )z + e7ReoT 72 1-2 Ree” zt + 7 Retz? 
— 0.9438 + 0.1604z7 4 0.9438 — 0.0017z 
1-0.7937z1+0.4576z7 1-0.7196z' + 0.151427 
0.0888z7! + 0.1749z7? + 0.0235z° 
~ 1,0000 - 1.513321 +1.1801z% -0.4495z* + 0.069327 


1—e%T 24 


The magnitude of the frequency response of the digital filter is plotted (in 
dB) in the following figure: 


0 O1 0.2 03 04 05 06 0.70809 1 
w/Tt 
We observe that the specifications are satisfied. The overlapping problem 


(as a result of the use of the impulse invariant method) is not apparent, here, as 
the analog filter is of limited bandwidth. 
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b. The bilinear method 
The analog frequencies are computed as follows: 


Os = 2.3417, 
2 


(0) 
Q,= 2tan{ = 0.8284, Q, = 2tan( 


For N, it is true, as before: 


Therefore, the required order for the Butterworth filter is N = 3 and is less 
than that required in the impulse invariant method. The frequency, Q,, is 
derived as follows: 


co 


4 com 0.8284 
= 0" a1)" ~ 07043 


= 1.1762. 
and it satisfies the second condition: 
| H, (jQ,) = 0.0158 < 0.0316 = 10°"? 


Thepoleslocatedinthelefthalfplaneare: s, =—0.5881 + 1.0186,s, ,s, =—1.1762. 
The transfer function of the analog filter is: 


QN 1.6272 
H,(s)= : = : 
on a (s? -2Re(s,)s+|s \(s—s,) 
1.6272 


~ (62 +1.1762s + 1.3834)(s + 1.1762) 


Finally, we apply the bilinear method: 


_ 0.0662+ 0.198721 + 0.1987z* + 0.0662z° 


H(z)=H (s = a eee 
(2) = FalS),-20-2tyase 1 -0.9359z1! + 0.5673z7 -0.1016z% 
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The magnitude of the frequency response of the resulting digital filter is 
illustrated in the figure below. We notice that the specifications are satisfied. In 
addition, although of lower order, the resulting filter has a response much closer 
to the ideal than the filter we designed using the impulse invariant method. 


20 


(@) 01 O02 03 04 05 06 0.708 09 1 


w/Tt 


6.179 Designa digital differentiator using the Hamming window. 


SOLUTION: 
Step 1: The desired frequency response of the analog differentiator, H(r), is defined: 
H(s) = $= A(r)= j |g on p= j20 fr 
ferfs 


Step 2: h[nT] is computed. 
1/2 1/2 
nnT|= | H(rjeP*"dr= | (jm frye" ar 
-1/2 1/2 
1/2 ; 1/2 f 1/2 ; 
| H(r)e""" dr = | (j2n f.r)e* " dr= 4 jan nre””™™ dr 
-1/2 -1/2 nN J-1/2 


1/2 =1/2 v2, 
oe rdei* my = El eat ar |"! -| olan “atn) 
n n s 


-1/2 re1/2 1/2 


f 1 r=1/2 
— Js re el? "| 


n j2an 


r=-1/2 


—ds 1 i elmn ie 1 a en 
n|2 j2nn n| 2 j2an 


= Sijein n +e "|= Bae [e” n eit a | 


fi 


j2nn? 


ss Te og n- [2jsinz n] 
2n 
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Thus, 


fe iy’ £2} sine" 1)" n#0 
27n n 


h[nT]=4 ” J 
fe A, ~sinan = f.[an— unt) 
m™ 
—l[zn-sinzn] 
= f lim dn 
* 130 dan? 
dn 
aie me Eos ant 
2 ney mn 
_ fey. Ua -acosan) / dn 
2 n>0 d(mn) / dn 
2. 
_ Sein m sina _ 4 —, 
2 n30 Nu 
fs (-1)" n#1 
h[nT]= 5 n 
0 n=0 


Step 3: Design of the filter using the Hamming window (M=7). 


0.544+0.46cos™™ — |n| <M 
Hamming window : w,[1] = M 


0 in| >M 


n=-1 


H(z)= Dies <(-1)"w, inte +E -(-1)" 0, Inlz" 


n= 7M 


AH @=e" He) 


The frequency response of the digital differentiator is illustrated in the follow- 
ing figure. 
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0.3 , 
7 
7 
Response with 
Rectangular window Ideal 

w 0.2 
wn 
Cc 
fo) 
a 
v 
wo 
=o) 
P= 
a 
= 
<x 

0.1 


Response with 


; : —> 
Hamming window 


(6) 0.1 0.2 0.3 0.4 0.5 


Normalized frequency =f/fs 


6.17.10 Design, using the window method, an FIR filter of linear phase and of N = 24 order 
that approximates the following magnitude of the ideal frequency response: 


(pie) —J1 |o|<0.27 
Hie = {4 0.22<|a|<z 


SOLUTION: We want to approach a low-pass filter with a cut-off frequency , = 
0.22. As N = 24, the frequency response of the filter to be designed is of the form 


H(e) = }° hme”, 
n=0 


Therefore, the delay of h[n] is N/2 = 12, and the ideal impulse response is 
sin(0.27(n-12)) 
ni|= 
m(n—12) 
To complete the design, we have to choose the window function. With a fixed 
window length, we ‘move’, during the design, between the width of the transition 
band and the ripple-width in the pass band and stop band. 


By using a rectangular window, which results to the smallest width of the tran- 
sition band, we get: 


i 


A@ = 27 - wa = 0.0752 
24 
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and the filter is 


eis sin(0.22(n— 12)) 
Ve i212) 


for Osns24 and h[n]=0 elsewhere 


The stop band attenuation is only 21dB, resulting in a ripple of 5, = 0.089. 
Using the Hamming window, the filter obtained is: 


) for 0<sns<24 


h[n] = oss 0.46c0s{ eu ] neat 2 


24 m(n—12) 


The attenuation of the stop band is 53dB, resulting in a ripple of 5, = 0.0022. 


The width of the transition band is consequently quite large: w=27- ae = 0.2752 


6.17.11 Design a FIR high-pass filter with the minimum possible filter order using the 
window method and having the following characteristics: stop band cut-off fre- 
quency of w, = 0.22z, pass band cut-off frequency of , = 0.282 and a stop band 
ripple of 5, = 0.003. 


SOLUTION: The stop band ripple 6, = 0.003 corresponds to the stop band attenuation 
a, = —20 log 6, = 50.46. 

For a filter of the minimum order, we use the Kaiser window with parame- 

ter, B, that is equal to # = 0.1102(a, - 8.7) = 4.6. Since the transition band width is 


Aw = 0.062 = 0.03, the required window width N = ven 


By choosing N = 99, a linear phase Type II filter is derived, the transfer function 
of which will have a zero at z = —-1. However, in this way, the frequency response 
gets to zero for w = a, so the value is not acceptable. We, therefore, increase the 
order of the filter to N = 100 to produce a linear phase Type I filter. 


= 98.67. 


To have a transition band that extends from @, = 0.227 to o, = 0.282, we set the 
cut-off frequency of the ideal high-pass filter equal to the average value of these 


0, +O, 


two frequencies, i.e., @, = =0.257. 


The impulse response of an ideal high-pass filter with zero phase and the cut- 


_ sin(0.2577n) 
mm 


off frequency w, = 0.251, is h,,,[n] = 6[1] , where the second term cor- 


responds to a low-pass filter with a cut-off frequency w, = 0.25z. In introducing a 
delay in h,,,[n] equal to N/2 = 50, we obtain: 


sin(0.252(n — 50) 
m(n—50) 
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The high-pass filter derived is h[n] = h,[n] - w[n], where w[n] represents the Kaiser 
window with N = 100 and f = 4.6. 


6.17.12 Design a digital, of linear phase, FIR filter using the window method that satis- 
fies the following specifications: 


0.99<|H(e”)|<1.01, 0<\ol<0.3” 


IH(e”)|<0.01, 0.35 <|ol< 


SOLUTION: Since the pass band ripple and the stop band ripple are the same, we can 
focus only on the requirement of the stop band ripple. A stop band ripple 6, = 0.01 
corresponds to a stop band attenuation of -40dB. Therefore, based on Table 6.2, we 
can use the Hanning window for which an attenuation of approximately -44dB is 
obtained. The transition band, according to the specifications, is Aw = 0.052 > Af= 
0.025. 


The required filter order is N= 7 =124, and it is 


w[n] = 0.5- 0eos( 2%") for O<n<124. 
124 
For an ideal low-pass filter with a cut-off frequency defined by the point 
corresponding to the middle of the transition band, i.e., @, =0.3252, and such a 
delay that hn] is symmetrically placed in the interval 0 <n < 124, that is N/2 = 
62, it is: 


ht] = sin(0.3252(n — 62)) 
| m(n— 62) 


Thus, the filter corresponds to 


, Osns124 


h[n]= [0s iBacs 2a sin(0.3252(n = 62)) 
i24 n(n — 62) 


If, instead of the Hanning window, we used the Hamming or Blackman win- 
dow, we would exceed the threshold of the ripple requirements in the pass band 
and the stop band, as a larger filter order would be required. For example, using 
the Blackman window, the order of the filter, which satisfies the transition band 


specification would be N = a 220. 
0.025 


6.1713 Design a digital IIR low-pass filter, which satisfies the specifications: 
H(e)) >-1dB for |w| <042 and Hie) <-15 dB for 0.57 < |w|<z 
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SOLUTION: Our design will be based on the Butterworth analog filter, and the final 
digital filter will have the corresponding form in terms of its response to frequencies. 


The frequency response of the Butterworth analog filter is given by: 


(ja? = + 


Q 2N 
1+ (2) 
Q, 
For example, for Q, = 100 and N=4, the filter will be of the form: 


W=0:0.1:200; 


N=4; 
Wce=100; 
H2=1./(1+(W/We) .*(2*N)); 
plot (W,H2) 
1 
0.8 ] 
0.6 7 
0.4 1 
0.2 1 
0 , 
0 50 100 150 200 


|H( ja); gets to the half of its maximum value for Q = Q.. 
e First step: Define and understand the specifications. 


Our design will be based on the Butterworth analog filter. Because of the 
monotony of the filter, the relations concerning the specifications are trans- 
formed equivalently into: 


eae 

H(e)| =-1dB for w=0,=04n 
3. 12 

Ate”) =-15dB for w=a,=0.5n 


¢ Second step: Transform the specifications into analog. 


We already mentioned that, as an intermediate design step, we will create 
an analog filter. For this reason, our specifications should be transformed into 
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the corresponding analog ones. Using the method of bilinear transform, the 
conversion from digital to analog will be accomplished. 


The relation of the bilinear transform which connects the analog Q with the 


digital ones @ is: Q= * tan( 2 Let the sampling period T = 1 sec. 


Thus, the specifications are now transformed as follows: 


H(jQ)? =-1dB for Q=Q, and Q,=*tan| @ 
J 1 1p 9 


lH(ja)" =-15dB for Q=Q, and Q,= 7 tan| 2 
J T 


2 


We now write the following code in MATLAB: 


oe lee 

wl=0.4*pi; 
w2=0.5*pi; 
W1=2*tan(wl1/2) ; 
W2=2*tan(w2/2) ; 


e Third step: Define N, Qc. 
The new system based on the new analog specifications is of the form: 


ilps —— sa SSR 


and 


where for the specific design R, = 1, A, = 15 
By solving the system, we can compute N: 


ny = 280 0%"? - 1/01) ] 


Q 
21og 19 a 


2 


Design of Digital Filters 347 


Because the resulting N is usually decimal, we round it to the next integer value. 
To determine Q., we have the following options: 


Q, Q, 


afta aay GO) 


We use the first of the above relations in our code: 


Q = 


ec 


Rp=1; 

As=15; 

numN=1og10 ((10* (Rp/10) -1) / (10% (As/10) -1)); 
denN=2*10g10 (W1/W2) ; 

N=ceil(numN/denN); % ceil rounds to the next integer 
denWe= (10* (Rp/10) -1) *(1/(2*N) ) ; 

We=W1/denWc; 


Fourth step: Define the poles of the analog filter. 

If we run the code, we find that N=8. Laplace transform will be of the form: 
QN 
(S—S,)...(S— Sg) 2N 
are the roots of the equation 1+ | =0 that have a negative real part. 

J 


c 


H(s)= , where s; represents the poles of the filter. These poles 


In our code, we will find the roots of this equation, and then we will keep 
the N of them that have a negative real part. This is because we want to make 
sure that our system is casual and stable. The MATLAB code is: 


Design the polynomial 
denanal=[(1/(j*Wc))*(2*N) zeros(1,2*N-1) 1]; 
polesanal=roots (denanal) ; 
polesanalneg=polesanal (real (polesanal) <0) ; 


e Fifth step: Apply the bilinear transform. 


= el 
By setting s= a, and after some calculations, we define the final digital 
z 


filter. By using the following MATLAB function, we can make the process described 
above very simple. So, we add the next line to our code: 


[Zdig, Pdig, Kdig] =bilinear([],polesanalneg,Wc*N,1/T) ; 


The bilinear function accepts as input arguments: the analog filter zeros (not 
present in Butterworth therefore []), the analog filter poles (polesanalneg), the 
multiplication constant (Wc‘N) and the sampling frequency (1/T). We run it (so 
the bilinear transform is applied), and the output will be: the digital filter zeros 
(Zdig), the digital filter poles (Pdig) and the multiplication constant (Kdig). 
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So the filter obtained will be of the form: 


pp, 2-40) 6-4) 
H(z) = Kaig Ele zy) 


where Kdig is real. Due to rounding errors MATLAB returns an extremely small 
fantastic part. For this reason, we add the following line in our code: 


Kdig=real (Kdig) ; 


e Sixth step: Define the digital filter coefficients. 


At this point, we have already fully defined the digital filter. If we were just 
asked to determine it, we would have finished. But we are interested in seeing the 
filter operate and in practice to filter out some sinusoidal components. In MATLAB 
this process is done with the filter function, which however requires knowledge 
of the filter coefficients. So we have to bring H(z) in the form of a quotient of two 
polynomials with respect to z". The coefficients of this polynomial are the coef- 
ficients we have to define. The code is: 


b=Kdig*poly(Zdig); % We compute the nominator coefficients 
a=poly (Pdig) ; % We compute the denominator coefficients 
b=real (b) ; 

a=real (a); 


Since filter function requires a(1) to equal to one, we add the following lines to 
the code: 


b=b/a(1) ; 
a=a/a(1); % Divide the numerator and the denominator with a(1) 


In order to design the frequency response, we use the commands: 


H=freqz(b,a, [0:0.01:pi]); 
plot ([0:0.01:pi],abs(H) .* 


2) 


0.8 


0.6 


0.4 


0.2 
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We can check whether the original digital specifications are satisfied using the 
commands: 


Hwlw2=freqz(b,a, [wl w2]); 
10*10g10 (abs (Hw1lw2) .*2) 


ans = 
-1.0000 -16.4299 


We note that, with respect to the first frequency, the initial condition is satisfied, 
whereas for the second frequency, there is a difference. This is due both to the 
rounding we made when choosing N and to the equation we chose to calculate Qc. 
The (unavoidable) difference in the second condition should not disappoint us. We 
were asking for a filter that would have an attenuation of 15 dB at ,. We finally 
designed a filter that has more attenuation, i.e., 16.4299 dB, at the same frequency. 
So we designed a better filter than the one the original specifications required. 


We will filter now two sinusoidal functions, one of which is in the pass band 
and the other in stop band. The code is as follows: 


n=0;: 200; 
xl=sin(0.1*pi*n) ; 
x2=sin(0.7*pi*n) ; 
xX=K14+xX2; 
y=filter(b,a,x); 
subplot (211) 
stem(n,x) 

subplot (212) 
stem(n,y) 


0 20 40 60 80 100 120 140 160 180 200 
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Using the following code, we can plot graphs with respect to frequencies: 


X=fft (x); 

Y=ff£t (y) ; 

subplot (211) 

plot (n* (2*pi) /length(x) , abs (X) ) 
subplot (212) 

plot (n* (2*pi) /length(y) ,abs(Y) ) 


6.17.14 Develop the FIR design process of digital filters using MATLAB. 


SOLUTION: The basic idea behind successfully designing FIR filters is to choose an 
ideal filter (which has always a non-causal and of infinite-length impulse response) 
and to cut off the impulse response to make the filter an FIR one. Consider an ideal 
linear phase filter with bandwidth o, < x 


H,(e@) = 1-e™ lols, 
: 0,0, <\al<a 


where w, represents the filter cut-off frequency. 
The infinite-length impulse response of the filter is: 


2 sin| @, (n- a) | 


h,[n] n(n —a) 


n=0,+1,+2,--- 


We create the function ideal_lp(wc,N) for the computation of the impulse 
response of the filter we described. 
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function [hd,n]=ideal_lp(wc,N) 
a=(N-1)/2; 

m=[0:1:(N-1)]; 

n=m-a+eps; 
hd=sin(we*n)./(pi*n); 


We then run it for N=21 and for a cut-off frequency of 0.257 to get the impulse 
response: 


{hd,n]=ideal_1p(0.25*pi,21); 
stem(n,hd) ; 


0.25 


Now, in order to implement an FIR filter, we must: 
a. Approximate hd[n] with a sequence h'[n] of finite length, N, using a window. 


b. Shift h’[n] in time by 1) = (N — 1)/2 samples, h[n] = l’[n — no], so as h[n] = 0,n < 0, 
for the filter to be casual. This is the equivalent of considering a = np. 


The most common windows are described by the following equations: 


<ini< N- 
RECTANGULAR w[n]= 1,0s\inlsN-1 
0,|n|>N 
2 pene 
BARTLETT w[n]= N y) 
aot Nal open i 
N’ 2 
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HAMMING — w[n]=0.54- 0t6cos{ 2") 9 <n<N-1 


In order to create these windows, we use the following MATLAB commands: 


W=boxcar (N) 
W=Bartlett (N) 
W=hamming (N) 
W=hanning (N) 


To see the effect of a rectangular window of length, N, ona FIR filter with @, = 
0.25x, we write the code: 


we=0.25*pi; 

N=11; 

n=0:1:N=1; 
hd=ideal_lp(wc,N) ; 
w_tet=(boxcar(N))'; 
h=hd.*w_tet; 
fr=0:0.01:pi; 
H=freqz(h,1,fr) ; 
figure 

subplot (121) 
semilogy (fr,abs (H) ) 
subplot (122) 

plot (fr,abs(H) ) 
N=21; 

n=0:1<:N=1; 
hd=ideal_lp(wc,N) ; 
w_tet=(boxcar(N))'; 
h=hd.*w_tet; 
fr=0:0.01:pi; 
H=freqz(h,1,fr); 
subplot (121) 

hold on 
semilogy(fr,abs(H),'k:') 
subplot (122) 

hold on 

plot (fr,abs(H),'k:') 
N=31; 

n=0:N=1; 
hd=ideal_lp(wc,N) ; 
w_tet=(boxcar(N))'; 
h=hd.*w_tet; 
fr=0:0.01:pi; 
H=freqz(h,1,fr); 
subplot (121) 
semilogy(fr,abs(H),'r--') 
subplot (122) 

plot (fr,abs(H),'r--') 
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10° 


10° 


0 2 4 0 2 4 
The phase of the filter is linear. We can design it for a frequency range, ie., 


[-0.157, 0.157], using the following commands: 


w=-0.15*pi:0.01:0.15*pi; 
H=freqz(h,1,w) ; 
plot (w,angle (H) ) 


4 T T T 


3P 4 


-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 


We notice that: The lobe frequency increases with N and the width of the 
first lobe remains almost constant. We also notice that Gibbs ringing has been 
introduced. 


Using the following code we obtain the response of the low-pass filter for N=21, 
and for the rectangular and Hanning windows: 


we=0.25*pi; 

N=21; 

n=0:1:N-1; 
hd=ideal_lp(wc,N) ; 
w_tet=(boxcar(N))'; 
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h=hd.*w_tet; 
fr=0:0.01:pi; 
{H,w]=freqz(h,1,fr); 
figure 

subplot (1,2,1); 
semilogy (fr,abs (H) ) 
subplot (1,2,2); 

plot (fr,abs (H) ) 
N=21; 

n=O; 1:N=15 
hd=ideal_lp(wc,N) ; 
w_han=(hanning(N) )'; 
h=hd.*w_han; 
fr=0:0.01:pi; 
[H,w]=freqz(h,1,fr); 
subplot (1,2,1) 

hold on 
semilogy(fr,abs(H),'r') 
subplot (1,2,2) 

hold on 

plot (fr,abs(H),'r') 
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10° 


We notice that: 
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1. The lobes, including the main, have a much smaller width for the Hanning 


window. 


2. Passing from the pass band to the stop band is much faster in the case of the 
rectangular window. This results from the fact that the main lobe of the Fourier 
transform of the Hanning window is of larger width. 


Next, we study an application of real-case filtering. Let a signal consisting of a 
sum of two cosines with frequencies of 30Hz, 50Hz and a sine of 60Hz. We want 
to apply a filter that cuts the frequencies higher than 40Hz. 
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£s=300; 
n=0:1/fs:0.2; 
X=COS (2*pi*30*n) +cos (2*pi*50*n) +sin(2*pi*60*n) ; 
we=pi*40/(fs/2) ; 
N=21; 

n=O; 1<N-=1; 
hd=ideal_ lp(wc,N) ; 
w_tet=(boxcar(N))'; 
h=hd.*w_tet; 
y=filter(h,1,x); 
fr=0:0.01:pi; 
Xw=freqz(x,1,fr 
Hw=freqz(h,1,fr 
Yw=freqz(y,1,fr 
subplot (311) 
plot (fr,abs (Xw) 
subplot (312) 
plot (fr, abs (Hw) 
subplot (313) 
plot (fr, abs (Yw) 


40 T T T T T T 
207 7 
@) 1 1 it 
0 0.5 1 1.5 2 2:5 3 3:5 
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6.17.15 Create functions in MATLAB to calculate the frequency response amplitude of 
linear phase FIR filters of Type I and III. 


SOLUTION: 
a. Type I linear FIR filters 
(M-1)/2 
In this case: H(e”) = | » a(n) cos «| eure 
n=0 
where sequence a[n] is derived from h[n] as follows: 


a(o) = r[ aoe 1) : the central sample 


a(n) =21( Mtn ; tens 


2 
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(M-1)/2 
Thus, itis H,(w)= » a(n)coswn 
n=0 
The function below specifies the amplitude of the frequency response of the FIR, 
linear, Type I filter. 


function [Hr,w,a,L]=Hr_Typel (h) ; 
[Hr,w,a,L]=Hr_ Typel (h) 
Hr = Amplitude Response 
w = 500 frequencies between [0,pi] over which Hr is computed 
%$ a = Type-1 LP filter coefficients 
L = Order of Hr 
h = Type-1 LP filter impulse response 


M length (h) ; 

L= (M-1)/2; 

a = [h(L4+1) 2*h(L:-1:1)]; % 1x(L+1) row vector 
n = [0:1:L] % (L4+1)x1l column vector 

w = [0:1:500]’*pi/500; 

Hr = cos(w*n)*a'; 


b. Type Hl linear FIR filters 
In this case: 


(M-1)/2 


H(e”)= »y c(n)sin wn |e 


n=1 


-|Z-wor-nr2| 


where the sequence c[n] is derived from h[n] as follows: 


(M-1)/2 
Thus, itis H,(w) = Dy c(n)sin wn 
n=1 
Note that, at w = 0 and w = x it is H,(w) = 0, regardless of c(n) or h(n). It is also 
el"? = j, which means that jH,(w) has only an imaginary part. This type of filter is 
not suitable for the design of high- and low-pass filters. This behavior is suitable 
for differentiators and ideal Hilbert transformers. 


Using the following function, the amplitude of the frequency response of Type 
IIL, linear, FIR filter is derived. 


function [Hr,w,c,L]=Hr_ Type3(h) ; 

% [Hr,w,c,L]=Hr Type3 (h) 

% Hr = Amplitude Response 

frequencies between [0,pi] over which Hr is computed 
Type-3 LP filter coefficients 

Order of Hr 


Sow 
% b 
’ L 
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% h = Type-3 LP filter impulse response 
M = length(h) ; 

L = (M-1)/2; 

ec = [2*h(L41:-1:1)]; 

m= [Ore] 

w = [0:1:500]’*pi/500; 

Hr = sin(w*n)*c’; 


Type II and IV filters (FIR and Linear phase) are studied next. 
Type I linear FIR filters 
In this case, it is: 


(M)/2 
H(e”)= | > Hopeos| of - a aaa 


n=1 
where the sequence a[n] is derived from h[n] as follows: 


b(n) = 2u( - n| , N= (ae 
2 2 


(M)/2 
Thus, H,(w)= > sr)eos|o{n-4)} 


(M)/2 


For w =, itis H,(w)= > b(n) cos| (1 - | =0 regardless of b[n] or h[n]. 


n=1 


This Type is suitable for high- and low-pass filters. 
Type IV linear FIR filters 
In this case, it is: 


sr{ Samm fao-2) Em 


n=1 


where the sequence d[n] is derived from /[n] as follows: 


M 


M 
—-N — 
2 2 


an) = 21 2 Mead 


(M)/2 
Thus, H,(w)= by aonsin o{n-3)| 


Note that, at w = 0 and w = 17, it is H,(0) = 0 and e*? = j. 


n=1 
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This behavior is also suitable for differentiators and approximation of ideal 
Hilbert transformers. 


6.17.16 Design the digital pass band filter with the following specifications: 


lower stopbandedge: w,,=0.2n%, A,=60dB 
lower passband edge: W,,=0.35"% R,=1dB 


upper passband edge: wy, =0.652, R,=1dB 
upper stopband edge: wy, =0.8z A, = 60dB 


The specifications above are illustrated in the following figure. 


0 0.35 0.35 0.65 0.8 1 
7 H i > w/pi 
1y— 
2) 
o 
2 
[S} 
Vv 
fa 
60 
vy 


SOLUTION: The requested code in MATLAB for designing the digital pass band filter is: 


wsl = 0.2*pi; wpl = 0.35*pi; 
wp2 = 0.65*pi; ws2 = 0.8*pi; 
As = 60; 


tr width = min((wpl-ws1), (ws2-wp2) ) ; 
M = ceil(11*pi/tr_ width) +1; 
n = [0:1:M-1]; 


wel = (wsl+wpl)/2; we2 = (wp2+ws2)/2; 
hd = ideal_lp(wc2,M) - ideal_lp(wc1,M) ; 
w_bla = (blackman (M))'; 


h = hd.*w_bla; 

[db,mag,pha,grd,w] = freqz m(h, [1]); 
delta_w =2*pi/1000; 

Rp -min (db (wp1/delta_w+1:1:wp2/delta_w)) % Actual Pass band Ripple 
As 


% Plots 
subplot (2,2,1); stem(n,hd); title('Ideal Impulse Response') 
axis([0 M-1 -0.4 0.5]); xlabel('n'); ylabel('hd(n) ') 


subplot (2,2,2); stem(n,w_bla); title('Blackman Window' ) 
axis([0 M-1 0 1.1]); xlabel('n'); ylabel('w(n)') 


subplot (2,2,3); stem(n,h); title('Actual Impulse Response') 
axis([0 M-1 -0.4 0.5]); xlabel('n'); ylabel('h(n)') 


-round (max (db (ws2/delta_w+1:1:501)) ) % Min Stop band Attenuation 
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subplot (2,2,4); plot(w/pi,db); axis([0 1 -150 10]); 
title('Magnitude Response in db') ;grid; 
xlabel('Frequency in pi units'); ylabel('Decibels') 
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In this code, the function freqz_m was used, which returns the amplitude of the 


frequency response in dB, the phase response and the group delay response. 


function [db,mag,pha,grd,w] = freqz m(b,a) 

% [db,mag,pha,grd,w] = freqz_m(b,a) 

% db = relative magnitude in dB computed over 0 to pi radians 
$ mag = absolute magnitude computed over 0 to pi radians 

% pha = Phase response in radians over 0 to pi radians 

% grd = Group delay over 0 to pi radians 


% w = 501 frequency samples between 0 to pi radians 
% b = numerator polynomial of H(z) (for FIR: b=h) 
% a = denominator polynomial of H(z) (for FIR: a=[1]) 
{[H,w] = freqz(b,a,1000,'whole'); 
H = (H(1:1:501))'; w= (w(1:1:501))'; 


mag = abs(H); 

db = 20*1l0g10((mag+eps) /max (mag) ) ; 
pha = angle(H); 

grd = grpdelay(b,a,w) ; 


0 20 40 60 0 0.5 1 
n frequency in pi units 


jw, 0O<~wsn 


6.17.17 Let H,(e”)= represent the frequency response of an ideal 


-jw, -™<w~<0 


digital differentiator. Design a digital FIR differentiator using a Hamming win- 


dow of length 21. 


SOLUTION: Note that M is an even number, so « = (M-1)/2 is integer and h,(n) will be 
zero for all n. Here, M must be an odd number, and that would be the case of an FIR, 


linear phase, Type III filter. 
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The code in MATLAB for designing the digital FIR filter is: 


M = 21; alpha = (M-1)/2; 
n 


= O:1:Mel1; 
hd = (cos (pi*(n-alpha)))./(n-alpha) ; hd(alpha+1) = 0; 
w_ham = (hamming(M))'; 


h = hd.*w_ham; 
[Hr,w,P,L]=Hr_Type3 (h) ; 


% Plots 

% subplots(1,1,1); 

subplot (2,2,1); stem(n,hd); title('Ideal Impulse Response’) 
axis([-1 M -1.2 1.2]); xlabel('n'); ylabel ('hd(n)') 

subplot (2,2,2); stem(n,w_ham); title('Hamming Window' ) 

axis([-1 M 0 1.2]); xlabel('n'); ylabel ('w(n)') 

subplot (2,2,3); stem(n,h); title('Actual Impulse Response') 
axis([-1 M -1.2 1.2]); xlabel('n'); ylabel('h(n)') 

subplot (2,2,4); plot(w/pi,Hr/pi); title('Amplitude Response') ;grid; 
xlabel('Frequency in pi units'); ylabel('slope in pi units'); 
axis([0 1 0 1]); 

In the MATLAB code, above, the function ideal_Ip was used to calculate an ideal 


low-pass filter of impulse response /,(1). 


function hd=ideal_lp(wc,M) ; 

% [hd] = ideal_lp(wc,M) ; 

% hd = ideal impulse response between 0 to M-1 
%$ wc = cutoff frequency in radians 

%$ M = length of the ideal filter 


alpha = (M-1)/2; 

n = [0:1:(M-1)]; 

m =n - alpha +eps; % add smallest number to avoid division by zero 
hd = sin(wce*m) ./ (pi*m) ; 


Ideal Impulse Response Hamming Window 
: 1 
SG L I 
S e? t = 
3 Opets $ i t i E05 il I 
-1 0 eof! Itee 
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n n 
Actual Impulse Response Amplitude Response 
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o 
-4 = 6 
OD 
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n frequency in pi units 
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6.17.18 Create a MATLAB code for designing a low-pass filter using the equiripple 
method. 


SOLUTION: The MATLAB function firpm, which is based on Parks-McClellan algo- 
rithm, can be used to design a FIR filter. 


Syntax of firpm function: firpm(N,F,A,W) where N: the order of the filter, F: 
Frequency vector [Fl F2 F3 ...], A: desired amplitude for each frequency [A1 A2 
A3...], W: weighting factor for each band in order to succeed equalization. Each W 
value indicates how much emphasis will be given to reduce the error of the transi- 
tion or stop band with regard to other bands (i.e., pass or stop bands). The length 
of W is half of F. Example: If F=[ F1 F2 F3 F4] : Two bands: F1-F2, F3-F4 with F2-F3 
the transition band, W=[W1 W2]. 


If we want to design Hilbert transformers or differentiators, we can use the 
functions: firpm(N,E,A,W,Hilbert’) or firpm (N,E.AW, ‘differentiator’). 


The MATLAB code is: 
Define sampling frequency 
fs=40000; 

Define filter order 

N=40; 

Define cut-off frequency 

£1=10000; 

Define transition frequency, 500Hz 
ftr=500; 

Fl=f1/fs; 

Ftr=ftr/fs; 


$Compute h coefficients for the filter 

h=firpm(N, [0 2*F1l 2*(F1+Ftr) 0.5*2],[11 0 0],[1 1]); 
S$Compute frequency response 

[H,W] =freqz(h,1,256); 

Design in Hz 

df=(0:length(W) -1)*(fs/2) /length (W) ; 

plot (d£,20*1log10 (abs (H) )) ; 


10 
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6.17.19 Describe the process of designing an FIR filter using a rectangular window. 


SOLUTION: Suppose we want to calculate h[n] coefficients for an ideal low-pass filter 


with the cut-off frequency @, = Z Since the filter is low-pass, the coefficients h[n] 


willbe h[n]= n= /5) 
MATLAB code: 
n=-20:1:20; % N=41 
h=sin(n*pi/5)./(n*pi); 
stem(n,h) ; 
0.2 
0.15; 
0.1 
0.05 ; 
0 
-0.05 
-20 


The ideal frequency response for @, = = and N =41is illustrated in the follow- 
ing figure: : 


0 © ® 


For the finite length filter (FIR), we need to keep a finite number of the h[n] coef- 
ficients. This leads to truncation that distorts the original ideal low-pass function. 
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This truncation is best expressed by the use of a window. In the following figures, 
the window sequence wn] is illustrated, as well as its frequency response. 


Frequency domain 


Magnitude (db) 


0 Ol 02 03 O4 05 06 O07 O08 09 


Normalized Frequency ( rad/sample) 


The transform from IIR to FIR is done by multiplying the (infinite) ideal 
sequence h[n] with a rectangular window w[n] of finite length N or else, in the 
frequency domain, by convolving h() with a rectangular window, w(@). 


Multiplication : A[1] = higeglnJw[n] 


Convolution : 1(@) = Nigeq(@) * W(@) 


In the figures below, we can see the result of multiplying the ideal sequence, h[n], 
with the window, wn], and the convolution of the ideal hw) with w(@), respectively. 


364 Digital and Statistical Signal Processing 


0.2, 


0.15; 


0.1 


0.05] 


Magnitude Response (dB) 


Magnitude (db) 
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Normalized Frequency ( rad/sample) 


6.17.20 Design the frequency response of a 5-point low-pass moving average filter. 


SOLUTION: First suggested way: 


We shall compute the frequency response based on Y() = H(@)X@) by calculat- 
ing the DTFT of x[n — k] (k = 0,1,2...N — 1). The MATLAB code is: 


syms X w 

N=5; 

X1=X*exp(-j*w); %X1=DTFT{x(n-1)}. 
X2=X*exp (-j*2*w); %X2=DTFT{x(n-2)}. 
X3=X*exp(-j*3*w); %X3=DTFT{x(n-3) }. 
X4=X*exp (-j*4*w); %X4=DTFT{x(n-4) } 
Y= (1/N) * (X+X14+X2+X3+X4); ZOutput 


H=Y/X; Filter response. 

H=simplify (H) ; 

% Design of amplitude using ezplot command 
ezplot (abs(H),[0 pil); 


Design of Digital Filters 365 


(exp(4 imag(w)) abs(exp(w 1i) + exp(w 2i) + exp(w 3i) + exp(w 4i) + 1))/5 
1 


0.8 
0.6 
0.4 


0.2 


Second suggested way: Computation of the frequency response using: 
1-e 1% 
H(o)=(1/N) 
N=5; %Filter order. 
omega = 0:pi/400:pi; %. 
Hw = (1/N) * (1-exp(-i*omega*N) ) ./ (1-exp(-i*omega) ) ; 
plot (omega,abs(Hw)); Design of magnitude 
xlabel('Frequency w(rad/s) [0 - u] ); 
ylabel ('Magnitude') ; 
title (Frequency response m.a. filter') 


Frequency response m.a. filter 


Magnitude 
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fo) for) 
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Frequency w(rad/s) [0 - 7] 
We write the following code for unit conversion: 


fs=40000; Define sampling frequency fs. 

N=5; %Filter order. 

omega = 0:pi/400:pi; 

Hw = (1/N) * (1-exp(-i*omega*N) ) ./(1-exp(-i*omega) ) ; 

df=(0:length (omega) -1) * (fs/2) /length(omega); unit conversion of y-axis 
in Hz 

plot (df,abs(Hw)); Design magnitude. 

xlabel('frequency f (KHz)'); 

ylabel ('Magnitude') ; 

title('Frequency response of m.a. filter); 
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Magnitude 


0 0.5 1 1.5 2 
frequency f (KHz) x 104 


Third suggested way: Compute and design the frequency response using the 
FFT of h[n] 


Z 


= 5; %Filter order. 

hn =(1/N)* ones(1,N); %Define impulse response 

hF = fft(hn,1024); Define number of samples of FFT of hn 
plot (([0:511]/512)*pi, abs((hF(1:512)))); SPlot magnitude 
xlabel('Normalized frequency wo ,[0 - u]'); 

ylabel ('Magnitude') ; 

title('Frequency response of m.a. filter) ; 


Frequency response of m.a. filter 


Magnitude 


0 1 1 1 1 1 
0 0.5 1 1.5 2 25 3 3.5 


Normalized frequency w ,[0 - 7] 


6.17.21 Design a moving average filter with N =5 to smooth the noise of the input: 


x= [00.1 -0.20.30 -0.100.2 -0.10.1 -0200 -01001051120.9111.11 
0.8 1.10.8 1.10.9 10.8 0.50.1 —0.2 0.3 0 —0.10 0.2 —0.10.1 -0.200 —0.10 0.1] 
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367 
SOLUTION: 
N = 5; %Filter order. 
hn = (1/N)*ones(1,N); % Define impulse response hn=1/N. 
a=1; %Define a: a(1)*y(n) = b(1)*x(n) + b(2)*x(n-1) +... 
x= 00.1 -0.2 0.3 0 -0.1 00.2 -0.1 0.1 -0.2 00 -0.1 0 O. 1 
2 


10 

0.9 1 1 2.1 10.8 Lt 0.8 2.1 0.9 L Ove 0.5 0.1L =-0.2 0.3 

0.1 00.2 -0.1 0.1 -0.2 0 0 -0.1 0 0.1]; SDefine b: a(1)*y(n 
(1)*x(n) + b(2)*x(n-1) +... 

y = filter(hn,a,x); 

t = 1l:length(x); %. 


plot(t,x,'-.',t,y,'-'); Plot input and output. 
legend('Real signal+ Noise', 'smoothing of noise',2); 
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6.17.22 Design a FIR filter of length N = 15 with 0-10KHz frequency pass band and 
sampling frequency f, = 40KHz. 


SOLUTION: The distance between the samples is a 


= 2,66KHz. 
Then, we define the desired magnitude H(k): H(0 to 3) 


= 1 and H(4 to 7)=0. 
We use the following type (N=odd): 


h(i y= HO AS “Aire os{ 2H) ) 


The desired frequency response for the FIR filter of length N = 15 is illustrated 
in the following figure. 
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Magnitude 


Possible Frequency f 
choise (kHz) 


The code for design follows: 


N=16; % Filter order 

HO=1; % H(0)=1, since H(0to4) = 1 
Hk(1:4)=1; % H(1to4)=1, since H(0to4) = 1 
Hk(5:8)=0; % H(5to8)=0 

for n=1:N 

S=0; 


for k=1: ((N/2) -1) 

S=S+2*Hk (k) * (-1) *k*cos ((2*pi*k* (n-1)+pi*k) /N) ; 

end 

h(n) =HO/N+S/N; 

end 

fs=40000; % Define sampling frequency fs 
[H,W]=freqz(h,1,256); % Compute frequency response 
df=(0:255)*(fs/2)/256; %* Unit conversion of y-axis in Hz 
plot (df,abs(H)); % Design magnitude 

xlabel('Frequency f£, KHz'); 

ylabel ('Magnitude') ; 

title('F.I.R. Filter with the F.S. method N=16,Fc=10KHz,Fs=40KHz') ; 


F.I.R. Filter with the F.S. method N=16,Fc=10KHz,Fs=40KHz 


Magnitude 
o o 
BR fo) 


° 
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0 1 1 1 
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Frequency f, KHz x 104 
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6.17.23 Design a FIR filter of length N=20 of w, = 2/2 (normalized frequency) and sam- 
pling frequency f, = 40KHz. 


SOLUTION: We calculate the distance between the samples os =n/10. 


We, then, define the desired magnitude H(k): H(0 to 5) = 1 and H(6 to 10) = 0. 
We use the following formula (N=even): 


oot 


h(i) = = tN 2 1)! cos( 42H *2ak)) 


The design code follows: 


N=20; %Filter order. 
HO=1; %H(0)=1 
Hk(1:5)=1; *H(1to5)=1 


Hk (6:10)=0; % H(6to10) = 1. 
for n=1:N 
S=0; 


for k=1: (N/2)-1 

S=S+2*Hk (k) * (-1) *k*cos ((2*pi*k* (n-1)+pi*k) /N) ; 

end 

h(n) =HO/N+S/N; 

end 

fs=40000; % Define sampling frequency fs 

[H,W] =freqz(h,1,256); % Compute frequency response 

plot (W,angle(H)); % Design phase 

xlabel('Frequency f, KHz');ylabel('Phase') ; 

title('F.I.R. Filter with F.S. method N=20,@c=n/2,Fs=40KHz') ; 


F.I.R. Filter with F.S. method N=20,w =1/2,Fs=40KHz 
4 T T T T T T 


Phase 


L 
0 0.5 1 1.5 2 2.5 3 3.5 
Frequency f, KHz 


plot (df,abs(H)); %* Design Magnitude 
xlabel('Frequency f, KHz') ;ylabel('Magnitude') ; 
title('F.I.R. Filter with F.S. method N=20,wc=n/2,Fs=40KHz'); 
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F.LR. Filter with F.S. method N=20,w,=n/2,Fs=40KHz 
1.2 T T T T T 


Magnitude 
oO 
roo) 


0 50 100 150 200 250 300 
Frequency f, KHz 


With the following commands, we can design the coefficients h(i): 


n=[1:N]; 
stem(n,h) ; 


0 5 10 15 20 


6.17.24 Design an FIR filter with the specifications of the following plot and with sam- 
pling frequency f, = 40KHz, for N=16, N=24, N=32. Design the phases for each N. 


Magnitude 


| Normalized frequency 


n/4 2(n/4) 3(n/4) T 


SOLUTION: We calculate the distance between the samples: 
For N=16, the distance between the samples is: 2 = 1/8; for N=24, the dis- 
tance between the samples is: = = 7/12; and for N=32, the distance between the 


samples is a = 1/16. 
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We define the desired magnitude, H(k): For N=16, it is: H(O to 2) = 1, H(3 to 4)=0, 
H6 to 8)=1; for N=24, it is: H(0 to 3) = 1, H(4 to 6)=0, H(7 to 12)=1; and for N=32, it is 
H(0 to 4) = 1, HG to 8)=0, HQ to 16)=1 

We use the following formula (N=even): 


Ax 


h(i) = oO NU 2- 1) eos{ EA2R | 


The following figure depicts the desired frequency response of a FIR filter for 
N=16, N=24 and N=32. 


iT} 
w 
N 


a m/12 Wg, Ray 


N=16 


Normalized frequency 


2(n/4) 3(r/4) Tt 


The design code is: 
N=16; %Filter order. 
HO=1; %H(0)=1. 
Hk(1:2)=1; %H(ltoc2) = 1 
Hk (3:4)=0; %H(3to04) = 0 
Hk(5:8)=1; %H(5to8) = 1 
for n=1:N 
S=0; 


S=S+2*Hk (k) * (-1) *k*cos ( (2*pi*k* (n-1)+pi*k) /N) ; 

end 

h(n) =HO/N+S/N; 

end 

fs=40000; Define sampling frequency fs 

[H,W] =freqz(h,1,256); sCompute frequency response 
plot (W,abs(H),'-.'); Design magnitude for N=16 order 
hold on 

N=24; %Filter order 
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HO=1; %H(0)=1. 

Hk(1:3)=1; %H(1to3) = 1 

Hk (4:6)=0; SH(4to6) = 0 
Hk(7:9)=1; %H(7to9) = 1 
Hk(10:12)=0; %H(10t012) = 0 
for n=1:N 

S=0; 


for k=1: (N/2)-1; 
S=S+2*Hk (k) * (-1) *k*cos ( (2*pi*k* (n-1)+pi*k) /N) ; 


end 

h(n) =HO/N+S/N; 

end 

fs=40000; 

{[H,W]=freqz(h,1,256); sCompute freq. response 
plot (W,abs(H),'--'); % Design magnitude for N=24 order 
hold on 

N=32; % Filter order 

HO=1; %H(0)=1 

Hk (1:4)=1; *H(1to4) = 1 

Hk(5:8)=0; %H(5to8) = 0 

Hk(9:12)=1; %H(9to12) = 1 

Hk (13:16)=0; %H(13t016) = 0 

for n=1:N 

S=0; 


for k=1:(N/2)-1; 

S=S+2*Hk (k) * (-1) *k*cos ((2*pi*k* (n-1)+pi*k) /N) ; 

end 

h(n) =HO/N+S/N; 

end 

([H,W] =freqz(h,1,256); sCompute frequency response. 
xlabel('Normalized frequency o,[0 to u]'); 

ylabel ('Amplitude') ; 

title('F.I.R. filter with F.S. method N=16 ,N=24 and N=32,Fs=40KHz') 


plot (W,abs(H),'-'); Design magnitude for N=32 order. 
legend ('N=16', 'N=24’ ,’N=32',3); 


t 


gral filter with F.S. method N=16 ,N=24 and N=32,Fs=40KHz 


r 
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0 0.5 1 1.5 2 25 3 35 
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6.17.25 Design a low-frequency FIR filter with the window method with the cut-off 
frequency f, = 10KHz, the sampling frequency f, = 40KHz, a transition band of 
725Hz, and a maximum loss of the gain in the transition band equal to 20dB. 
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SOLUTION: Based on the filter specifications, we choose the rectangular window, w(n), 
since the requested gain loss rate of the transition band is equal to 20dB. Thus, K=0.9. 


0.9 

We compute the order, N, of the filter using the formula N = 

ie, N 2>26,1 or N=27 (integer). 10.725 Kz - 10KH2 
40KHz 

We calculate the coefficients of the ideal impulse response of the filter h[n] 

obtained by the formula sin(n27F) th this way, the frequencies are transformed 
nn 
from analog (f) to normalized digital (F). 
fie at OE 956) oe oh on Os 
f, 40KHz 


We shift the ideal f[n] to the right by (N-1)/2 according to: 


We compute the FIR filter of the finite sequence h(n) by multiplying the ideal 
h{n| with the window w[n}: h(n) = hghiftea WM), 0 <n < (N-). 


The design code is: 


fs=40000; Define sampling frequency fs 

N=27; %Filter order 

w=ones(1,27); %Define rectangular window 
F1=0.25; %Fl=f1/fs, where £1 cut-off frequency 


w1=0.25*2*pi; Yw1=F1*2n 


M=(N-1)/2; %Shift by (N-1)/2 

n=-M:M; 

h=2*F1*sinc(2*F1*n); Compute the ideal impulse response h 
h=h.*w; SMultiply h coefficients with the window 
[H,W]=freqz(h,1,256); sCompute frequency response 
df=(0:length(W) -1)*(fs/2)/length(W); % Unit conversion of y-axis in Hz 
plot (d£,20*1log10 (abs (H) ) ) $Design magnitude in dB 

xlabel('Frequency f(KHz)' 

ylabel('Magnitude db'); 

title('Frequency Response of FIR filter (rectangular window) ') 


he 


1 


Frequency Response of FIR filter (rectangular window) 


Magnitude db 


-80 f ! f 
0 0.5 1 1.5 2 


Frequency f(KHz) x 104 
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What follows is an alternative way of writing the code that designs the requested 
filter: 


fs=40000; Define sampling frequency fs 

N=27; %Filter order 

h=firl(N,0.5,rectwin(N+1));%Compute h coefficients (rectangular window) 
[H,W] =freqz(h,1,256); sCompute frequency response 

df=(0:length(W) -1)*(fs/2)/length(W); % Unit conversion of y-axis (in Hz) 
plot (df,20*log10(abs(H))); *Design magnitude 

xlabel('Frequency f(KHz)'); 

ylabel('Magnitude dB'); 

title('Frequency response of FIR filter (rectangular window) 2™¢ way') ; 


Pe bald response of FIR filter (rectangular window) 2nd way 


Magnitude dB 


-120 1 1 1 
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Frequency f(KHz) x 104 


6.17.26 Design a band pass filter with frequency passing range from f, = 4,3KHz to f, = 
5KHz, transition band equal to 1.58KHz, in which the gain to be lost is at least 
45dB and the sampling frequency is f, = 15KHz. 


SOLUTION: Based on the filter specifications we want to design, we choose 
Hamming window w(n) and the desired rate of gain loss of the transition 
band equal to 45dB. Thus, K=3.3. The order of the filter N is computed by 
N= 4, 3KHz 43-1 5)KEE = 7 5. 31.32, ie, N > 31.32 or N = 32 (we choose 

15KHz 

a larger number). 


The coefficients of the ideal impulse response of the filter h[n] are derived using 
sin(n27F,)  sin(n27F,) 
nt nt 

log (f) to normalized digital (F). 


the formula . So, the frequencies are converted from ana- 
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fo: & aa E087 or @, = 2nF, =20* (4,3/15)=1.8 
: ba 
5KHz 
fpohih= Bae a1/3 or @, =20F, =22/3=2,094 


We shift the ideal [n] to the right by (N-1) according to: 


We compute the FIR filter of the finite sequence h(n) by multiplying the ideal 
h{n| with the window wn}: h(n) = lghifteq WM), 0 <n < (N-1). 


The design code is: 


fs=15000; Define sampling frequency fs 
£1=4300; Define fpassl 

£2=5000; Define fpass2 

Fi=f1/fs; Normalized F1 

F2=f£2/fs; tnormalized F2 

N=32; %Filter order 


w=hamming(N+1)!'!; Define Hamming window 
M=N/2; %Shift by N/2 
n=-M:M,s 


A=2*F1*sinc(2*Fl1*n); Compute the coefficients for low pass with Fcl=F1 
B=2*F2*sinc(2*F2*n); % Compute the coefficients for low pass with Fc2=F2 
h=B-A; %Subtract the Fcl coefficients from Fc2 

hh=h.*w; Multiply band pass coefficients with the window 
[H,W]=freqz(hh,1,256); sCompute frequency response 

df=(0:length(W) -1)*(fs/2)/length(W); % Unit conversion of y-axis (in Hz) 
plot (df,20*log10(abs(H))); *Design magnitude 

xlabel('Frequency f(KHz)'); 

ylabel('Magnitude db'); 

title('Frequency response of FIR filter (hamming window) 1° way'); 
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Ph elcid response of FIR filter (Hamming window) 1st way 


Magnitude db 
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What follows is an alternative way of writing the code that designs the requested 
filter: 


fs=15000; %Define sampling frequency fs 

N=32; %Filter order 

£1=4300; %Define fpass1 

£2=5000; %Define fpass2 

f=[f1 £2]/(fs/2); sDefine frequency range from f1 to £2 
h=firl(N,f,hamming(N+1));%Compute coefficients using the hamming window 
[H,W] =freqz(h,1,256); Compute frequency response 

df=(0:length(W) -1)*(fs/2)/length(W); % Unit conversion of y-axis (in Hz) 
plot (df,20*log10(abs(H))); *Design magnitude 

xlabel('Frequency f(KHz)'); 

ylabel('Magnitude dB'); 

title('Frequency response of FIR filter (hamming window) 2™2 way '); 


Pe quency response of FIR filter (Hamming window) 2nd way 


Magnitude dB 


0 1000 2000 3000 4000 5000 6000 7000 8000 
Frequency f(KHz) 


150KHz to f, = 250KHz, transition band equal to 50Hz in which the gain that 
will be lost is at least 60dB and with the sampling frequency f, = 1KHz 


6.17.27 Design a band stop filter using a Kaiser window with frequencies from f, = 
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SOLUTION: Based on the filter specifications, we choose the Kaiser window w(n) with y 
equal to y = 0.1102(A-8.7). From the specifications A = 60 and y = 0.1102(A-8.7) > = 5.65. 
We compute the order of the filter N. Since: 
179 ayn OIE 410 28 
14.36 


1000Hz 


A>50 >N= +1=73.49 or N =74. 


The coefficients of the ideal impulse response of the filter h[n] are derived using 


sient) Snes) So, the frequencies are converted from ana- 


the formula 
nm nt 


log (f) to normalized digital (F). 


150Hz 
fivzh: R= ‘ 1000Ez > 0.15 or @, = 27F, =2n* (150 / 1000) =0.9 
fob: h= ; = ate =0.25 or @, =2nF, =27.0.25 =1.57 
: z 


We shift the ideal h[n] to the right by (N/2) according to: 


We compute the FIR filter of the finite sequence h(n) by multiplying the ideal 
h{n| with the window w[n}: h(n) = hyjip.ao(), 0 <n < (N - 1). 
The design code is: 


fs=1000; % Define sampling frequency fs 
£1=150; %Define f1 

£2=250; %Define £2 

Fl=f1/fs; tNormalized frequency F1 
F2=f2/fs; % Normalized frequency F2 
N=74; %Filter order 


w=kaiser(N+1,5.67)!; Define window Kaiser (y=5.65) 
M=N/2; %Shift by N/2 
n==-M:M; 


A=2*F1*sinc(2*F1*n); Low pass coefficients with Fc=Fcl 
B=2*F2*sinc(2*F2*n); SLow pass coefficients with Fc=Fc2 

C=2*0.5* sinc(2*0.5*n); %Coefficients of all pass filter 

h=C- (B-A); %Compute coefficients of ideal stop band filter 
hh=h.*w; Multiply the coefficients with the window 

[H,W] =freqz(hh,1,256); sCompute frequency response 

df=(0:length(W) -1)*(fs/2)/length(W); % Unit conversion of y-axis (in Hz) 
plot (df£,20*log10(abs(H))); *Design magnitude 

xlabel('Frequency f£(Hz)'); 

ylabel('Magnitude dB'); 

title('Frequency response of FIR filter (kaiser window) 15* way') ; 
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36 Frequency response of FIR filter (Kaiser window) 1st way 


Magnitude dB 
& 
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What follows is an alternative way of writing the code that designs the requested 
filter: 


fs=1000; %Define sampling frequency fs 

N=74; %Filter order 

£1=150; %Define fpassl 

£2=250; *Define fpass2 

f=[f1 £2]/(fs/2); sDefine frequency range from f1 to £2 

h=firl(N,f, 'stop', kaiser(N+1,5.67));%Compute coefficients using 
the Kaiser window 

[H,W]=freqz(h,1,256); Compute frequency response 

df=(0:length(W) -1)*(fs/2) /length(W); % Unit conversion of y-axis (in Hz) 

plot (d£,20*log10(abs(H))); tDesign magnitude 

xlabel('Frequency f(KHz)'); 

ylabel('Magnitude dB'); 


title('Frequency response of FIR filter (Kaiser(y=5.67) window) 274 
way '); 


Preauenty response of FIR filter (Kaiser(y=5.67) window) 2nd way 


Magnitude dB 
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6.17.28 Design a band pass FIR filter using the equiripple method to allow the signal 
to pass at the frequency range 10KHz to 13KHz, with transition band at 1KHz, 


with a gain loss of 50dB and ripple in the pass band at 0.25dB (sampling fre- 
quency fs = 40KHz) 
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SOLUTION: The design will be implemented with the following MATLAB code: 


fs=40000; Define sampling frequency fs 

flp=10000; %Define fpassl 

ftr=1000; %Define transition width 

fls=flp-ftr; *Define fstopl 

£2p=13000; %Define fpass2. 

f2s=f2p+ftr; tDefine fstop2 

Flp=flp/fs; %sConverse to normalized frequency. 

Fis=fls/fs; 

F2p=f2p/fs; 

F2s=f2s/fs; 

Ftr=ftr/fs; 

f=[0 2*Fls 2*Flp 2*F2p 2*F2s 0.5*2]; % Define vector [Fl F2 F3 ..] 
m=[0 0110 0]; % Define desired width for each frequency 
wlp=2*pi*Flp; % Converse to normalized frequency w=2n(f/fs). 
wls=2*pi*Fl1s; 

Ww2p=2*pi*F2p; 

Ww2S=2*pi*F2s; 

wtr=2*pi*Ftr; 

Rp=0.25; %Define ripple Rp 

As=50; %Define As amplitude. 

dp=(10* (Rp/20) -1) /(10* (Rp/20)4+1); %Compute 5p 

ds=(1+dp) *(10*(-As/20)); %Compute 6s 

Weights=[1 ds/dp 1]; %Define weighting vector 

DW=wtr/ (2*pi) ; 

$Compute N (rounding) 

N=ceil( (-20*1lo0g10 (sqrt (ds*dp) )-13)/(14.6*DW) +1) ; 
h=firpm(N,f,m ,Weights); %Compute filter coefficients 

[H,W] =freqz(h,1,256); Compute frequency response 
df=(0:length(W) -1)*(fs/2) /length(W); % Unit conversion of y-axis (in Hz) 
plot (df,20*log10(abs(H))); *Design magnitude 
xlabel('Frequency f(KHz)'); 

ylabel('Magnitude dB'); 

title('Frequency response of FIR filter, method: equiripple '); 
It is computed that N=85. 


Frequency response of FIR filter, method: equiripple 
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Statistical Models 


The estimation, detection and classification processes can be grouped altogether under 
the broader title of statistical evaluation, which is the process inferring properties 
about the distribution of a random variable, X, given a particular state, x, which is called 
a data sample, measurement or observation. One basic idea is that of the statistical model, 
which is simply a hypothetical probability distribution or a density function, f(x), for the 
X variable. In general, statistics examine the possibility of matching a defined model in 
the set of x data. To simplify this task, f(x) can be limited to a class of parametric models 
{ f(x ;0)} aor Where f(x;8) is a known function and @ denotes an array of unknown param- 
eters that take values from the parametric space, ©. In this particular case, the process of 
statistical extraction shows the properties of the actual values of @ by the parameterization 
of f(x; ) that generated the sample x. 

In this chapter, we will discuss several models related to (a) the well-known Gaussian 
distribution, (b) the general category of exponential distributions and (c) the fundamental 
concept of statistical sufficiency on obtaining the properties of 0. 


7.1 The Gaussian Distribution and Related Properties 


The Gaussian distribution plays an important role in parametric statistics due to the rela- 
tively simple Gaussian model and its broad spectrum of applications. Indeed, in engineer- 
ing and science, the Gaussian distribution is probably the most jointly used distribution 
for random measurements. The Gaussian distribution is also called normal distribution. 
The probability density function (PDF) of a Gaussian random variable X (RV) is param- 
eterized by two parameters 0, and @,, which is the position parameter yu (« € R) and the 
(squared) scale parameter o7(o? > 0), respectively. The PDF of the RV x is given by: 


_(x-y? 


2 
@ 27  -~w <x <+00, 


fee = es 


When yp = 0 and o? = 1, X is called standard Gaussian RV. A Gaussian RV with position 
parameter y and scale parameter o? > 0 can be expressed as: 


X=oZ+U (7.1) 


where Z is the standard Gaussian RV. 
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The cumulative distribution function (CDF) of a standard Gaussian RV is denoted as 
N(z) and is defined as: 


N(z)=Pr[Z <z] 


v 


Z 1 ve 
= e 2 dv. 
la 


Using Relation 7.1, the CDF of a non-standard Gaussian RV X with parameters p and o? 
can be expressed in the form of the CDF of the standard Gaussian RV Z: 


Pax <a]=Pe[(X-w/o<e—w/o]=v {4 ) 


The CDF of the standard Gaussian RV, N(z), is associated with the error function 


erf(u) = = e dt,x>0, through the expression: 
0 


s[1+ arf (x\/V2)], x20, 


N (x)= 
slt-erflel/V2)], x<0. 


When v is a positive integer, the statistical moments of a standard Gaussian RV Z are 
given as: 


B[Z’]= (v—1)(v-3)---3-1, v is even 
0, v is odd 


foo 
where E| 9(Z) |= } g(z)f(z)dz denotes the statistically expected value of RV Z and f(z) is 
the PDF of Z. 

In particular, using Relation 71, the first and second statistical moments of a non- 
standard Gaussian RV X are E[X]=pu and E[X*]=y’+0”, respectively. Therefore, 
with respect to a Gaussian RV X, the expected (mean) value E[X]=y and variance 


var(X) = E| (X —E[X]) |=B[X?]-E?[X]=o? denote the position and scale parameters of 
the PDF f(x; , 0”) of X, respectively. The following relation will then be used for denoting 
the non-central mean dispersion E [|x + a | for a Gaussian RV X: 


[x +ai]= [20 +a(1-20'a)) 
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Also, the following notations are useful: 


¢ The RV X is Gaussian distributed with expected value yw and variance o*. Then, it 
is true that X ~ N(u, 0°). 


e The RV X is equal to a Gaussian RV that changes and shifts as follows: 


X=a Z +beX=N(b,a”’). 
N (0,1) 


Yvon = yx, 
i=1 i=1 


¢ The CDF of \(0, 1) is equal to a when checked at point 2: 
N(v) =a. 
e The inverse CDF of N(0, 1) is equal to « when checked at point v: 
N1(v)=a@. 


e Vector x is distributed as an n-dimensional vector with Gaussian RV, with expected 
values p and a covariance matrix R: 


x~ NM, R). 


7.1.1 The Multivariate Gaussian Distribution 


When a sequence of independent and identically distributed (IID) Gaussian RVs passes 
through a linear filter, the output remains Gaussian distributed but is no longer IID. The 
filter smooths the input and introduces a correlation between the values of the input 
vector. However, if the input sequence is a Gaussian RV, the output behaves the same. 
Consequently, the joint distribution of any p samples of the input sequence is a Gaussian 
multivariate. More specifically, a vector of random variables x = [X,,..., x is multivariate 
Gaussian with a vector of expected values p and a covariance matrix A, if and only if the 
joint PDF is in the form: 


fx) = eT AR op( ac wa'e-w), xeR’. (7.2) 


where | A | expresses the determinant of A. The p-variate Gaussian distribution depends 
on p(p + 3)/2 parameters, which we can reconstruct in a parametric vector 8 containing the 
p elements of the vector of expected values: 


y= [ey Mp] = iLx], 
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and on p(p + 1)/2 discrete parameters of the symmetric positive-definite covariance matrix 
ae 


A= cov(x)=E[(x-p)(x-p)]. 


e¢ Unimodality and symmetry of a Gaussian PDF: The multivariate Gaussian dis- 
tribution is unimodal (it has a single peak) and symmetric around its expected 
value. 

e The uncorrelated Gaussian RVs are independent: When the covariance matrix A is 
diagonal, i.e., when it is true that cov(x;, x)) = 0,14], then the multivariate Gaussian 
distribution is simplified as a product of the individual PDFs: 


: 
foo=] [ fe, 


where 


f(x)= 


210; 


the scalar Gaussian PDF with o? = var(x;) 


e The limit distribution of a Gaussian multivariate distribution remains Gaussian 
distributed: If x = [x,,---, x,,]" is a multivariate Gaussian vector, then every subset of 
the elements of x is a multivariate Gaussian vector. For example, the RV x, is scalar 
Gaussian, while vector [x,, x,] is bivariate Gaussian. 


e The linear combination of Gaussian RV is a Gaussian RV: Let the Gaussian multi- 
variate vector x =[x,,-:-,X,,]' and a matrix H p x m with constant (non-random) 
values. Then, the vector y = Hx consists of linear combinations of RV x;. The distri- 
bution of y is p-variate Gaussian with expected values = E[y]=H yp and covari- 


ance matrix p x p A, =cov(y) = Hcov(x)H’. 


e A vector IID with zero expected values remains unchanged in variations: Let the 
Gaussian multivariate vector x =[x,,---,X,,]' with zero expected values (= 0) and 
covariance matrix cov(x) = oI. If U represents an orthogonal matrix m x m, that is, 
U'U =I, then the vector y = U"x follows the same distribution of x. 


e The conditional Gaussian distribution, given another Gaussian distribution, is also 
a Gaussian: Suppose the vector z’ =[x",y"]=[x1,---,Xp)/Yir--/Y,] is a Gaussian 
multivariate ((p + q)-variate) with pw! = [ur at | expected values and covariance 
A,. Then, the conditional PDF f,,,(y|x) of y, given x, is a Gaussian multivariate 
(q-variate) of the form of Relation 7.2 with expected values of parameters and cova- 
riance matrix A: 


Hy (x) = Ely |x]=py + Ay Ay (x-B), (7.3) 
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and 


T -1 

Ay, = cov(y|x)= A, — A, AY A, y- (74) 

e The conditional expected value of a Gaussian multivariate, given another Gaussian 
RV, is linear, and the conditional covariance is constant: For the former multivari- 
ate vector, z7, the partitioned covariance matrix is: 


A, Ay, 
A, = 

Zz T y 
Nyy A, 


where A, =cov(x)=E[(x—-p,)(x-p,)" ] ispxp, A, =cov(y)=E| (y-H,(y-Hy)” | 
isqxqand A, =cov(x,y)= af (x- By - My)" | isp xq. 

The vector of the expected values of the conditionssal PDF f(y|x) is linear to x and 
is given in Equation 7.3. In addition, the conditional covariance does not depend 
on x and is expressed in Equation 74. 


7.1.2 The Central Limit Theorem 


One of the most useful statistical tools is the central limit theorem (CLT). The CLT allows 
for the approximation of the sum of the IID of the finite RVs with the use of the Gaussian 
distribution. What follows is a general version of the CLT that applies to RV data estimated 
by vectors. 

(Lindeberg-Levy) Central Limit Theorem: Let {x;}/_, IID random vectors in the R? plane with 
a joint expected value E[x;]=p and a finite positive covariance matrix A = cov(x,). In the 


case where n > + 00, the distribution of the RV vector z,, = ey ee —) converges to the 
i=1 

p-variate Gaussian distribution, with an expected value of zero and a covariance matrix A. 

Also, the CLT can be expressed as a function of the expected sample value 


Vn (x(n)-p) > z, n— +o, 


where z is a vector with Gaussian RVs, expected values of zero and a covariance matrix A. 
Therefore, for a large but finite range of values, vector x tightly approximates a Gaussian RV 


x ~(z/Vn+u), 


with an expected value p and covariance A/n. For example, in the case of a scalar value x, 
the CLT allows the useful approach for large n: 


n y 
: 1 oe) 
Pri} 1? x; Sy |= ex dy. 
| az / la P 20°/n ‘ 
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7.1.3 The Chi-Squared RV Distribution 
The central chi-squared PDF with k degrees of freedom is given by: 


_ 1 k/2-1,-x/2 
f.(x)= 2*PF(k/2) ~ e , x>0, (7.5) 


where k is a positive integer and I'(u) denotes the Gamma-function: 


oo 
T(u)= i x" le* dx, 


0 


For integer values of n, 


T(n+1)=n!=n(n-1)---1and (n+1/2)= eee en 


n 


If z, ~ NO), i = 1, ---, n, is IED, then => ie 


ri 


is distributed as chi-squared with n 
i=1 
degrees of freedom. The above case is denoted as: 


Yvan =z, (76) 


This approach, ie., expressing a RV in a chi-squared form, is also known as stochastic 
representation because it is expressed through operations reflected on other RVs. 
Some useful properties of the chi-squared distribution are: 


* Ely,l=n, var(y,)=2n. 
e Asymptotic relation for large n: 


x, =V2nN (0,1) +n. 


e The notation y, expresses an exponential RV with an expected value 2 and PDF 


= 1 —x/2 
f= per? 
. Vin is a Rayleigh distributed RV. 


7.1.4 Gamma Distribution 
The PDF is defined as: 


faa x ee, 20, 
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with parameters A, r > 0. Let {y;}", be the IID RV that follows the exponential PDF with an 
expected value of 1/A: 


A= oy, y>0. 


It is true that the sum x= Dy? follows the Gamma distribution. Other useful proper- 
i=l 
ties of the Gamma distribution of a RV x with parameters (A, r) are: 


e ElxJ=r/a. 


e varl[x] = 1/22. 


e The chi-squared distribution of a RV with k degrees of freedom is a special case of 
the Gamma distribution when: 4 = 1/2 and r=k/2. 


7.1.5 The Non-Central Chi-Squared RV Distribution 


The sum of squares of independent Gaussian RVs with unitary variance but non-zero 
expected values composes a non-central chi-squared RV. Specifically, if z; ~ M(wu;1) are 


n 


statistically independent RVs, then x= z? is distributed as a non-central chi-squared 
'y P i q 


i=1 iw 
RV with n degrees of freedom and a non-centrality parameter 6 = » u;. We usually use: 
i=1 


YiIvon+u) = YM DE = ano: (77) 


The noncentral chi-squared PDF cannot be expressed in a simple closed-type form. 
Therefore, some of its asymptotic features are listed below: 


¢ Ely,s]=nt+6, var(7,,5)=2(n+ 26). 


© 442,242 isa Rician RV. 


7.1.6 The Chi-Squared Mixed Distribution 


The distribution of the sum of squares of independent RVs, with zero-mean (expected 
value) but with different variances, is not expressed in a closed form. However, due to the 
importance of this distribution in a variety of practical applications, the following notation 
will then be used to express it: 


n 


» C; 2 _ 
Z; =X 
xe 
j 


i=1 


390 Digital and Statistical Signal Processing 


A useful asymptotic property is the following: 
2 
Ci 


ALZ,.]=1, varZ,.)=2>° yc 
i=l j 


J 


in = 1 ; ere 
Moreover, in the specific case where 7,,, =— 7, then the chi-squared mixed distribution 
“on 


is simplified in a scaled central chi-squared distribution. 


7.1.7 The Student’s t-Distribution 


For z ~ N(0,1) and y ~y,, independent RVs, the ratio x=z/,/y/n is called Student-t T,,, RV 
with n degrees of freedom. 
The PDF of 7,, is defined as: 


- T({n+1]/2) 
T(n/2)V20(1 $32 /nyrn , 


Fil) 


where 1 is a positive integer. Some important properties are presented next: 


° E[Z,]=0(n>1), var(T,)= re (n>2). 


e T,=N(0,1), n+. 
For n = 1, the expected value of 7, does not exist, and for n < 2, its variance is not finite. 
7.1.8 The Fisher-Snedecor F-Distribution 


Foru~y,,and v ~ y,, independent RVs, the ratio x = (u/m) / (v/n) is called Fisher/Snedecor-F 
(or Fisher-F) RV with m,n degrees of freedom: 


The Fisher/Snedecor-F PDF is defined as: 


x>0, 


V([m+n]/2) ee xl 2)/2 


f(x) = T(m/2)0(n/2) yi _ (m+n)/2 7 
4) 
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where , m positive integers. It should be noted that the statistical moments E[x*] of 
greater order than k = n/2 do not exist. A useful asymptotic relation is: 


oe ca Xm 


7.1.9 The Cauchy Distribution 
The ratio of independent RVs N(0,1) u and v is called the standard Cauchy distribution and 
is defined as: 


x=u/v ~C(0,1). 


The PDF is expressed as follows: 


—— xeR. 


~ m(1+x7)’ 


If @ = [y, o] the position and scale parameters (o > 0), then fy (x) = f((x — »)/e) denotes the 
shifted and scaled version of the standard PDF of the Cauchy distribution, C(y, 0”). 
Some notable properties are: 


e The Cauchy distribution does not produce statistical moments for positive integer 
order. 


¢ The Cauchy distribution is identical to the Student’s t-distribution with 1 degree 
of freedom. 


7.1.10 The Beta Distribution 


For u ~ y,, and v ~ y,, independent chi-squared RVs with m and n degrees of freedom, 
respectively, the ratio x = u/(u + v) is expressed by the Beta distribution: 


Xm __ = B(m/2,n/2), 
Xm +X) 


where B(p, g) represents a RV with a Beta PDF and parameters @ = [p, q]. It is of the form: 


f(x) xP t(1—x)t, xe[0,1], 


~ B(p,q) 


_fariq srt gy -PO@) 
where Bipa)= | (1-x)T dx= Tip+a)" 
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Some notable properties are: 


e¢ E[B(p,g)l=p/(p+9). 
° var(B(p, 9) = pq/((p + 9 + 1)(p + 9)’). 


7.2 Reproducing Distributions 


The RV x has a reproducing distribution if the sum of two independent x-variables, that is, 
x, and x,, follows the same distribution as the individual RVs, probably though with differ- 
ent parameters. The Gaussian RVs are described by the reproducing distribution: 


N (uy,07)+.N (uy, 63) = (4 + ty, 0? +03), 


which results from the fact that the convolution of two independent Gaussian RVs pro- 
duces a Gaussian RV. From Relations 7.5 and 7.7, it can be derived that the distributions 
of the central and the non-central chi-squared distribution are reproducing distributions: 


© Xt Xn =AXmsw lf X, and y, are independent RVs. 


e Xm, + Xn 65 = KX m+n,b,+5 7 if Xm,6, and Xn, are independent RVs. 


The mixture distributions of chi-squared, Fisher-F and Student-t have no densities. 


7.3 Fisher-Cochran Theorem 


The Fisher-Cochran theorem is useful for deriving the distribution of the squares of 
Gaussian distributed RVs. 

Let x =[x,,- oe a vector of RVs distributed by N(O, 1) and an idempotent matrix A of 
p order (i.e., A A = A). Then: 


x"Ax= 7, 


Proof: 
Let A= U AU’ be the eigendecomposition of A. Then, all the eigenvalues of A matrix 
are either 0 or 1: 


A A=UAU'UAU! 


=I 


=UNU' =UAU' 
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and, consequently: 


xTAx=x'UA U'x 
z=N,(0,1) 


-Saz-S [Lion 
i=1 i=l 


7.4 Expected Value and Variance of Samples 


Let x; be an IID RV \(0,1). The expected value and the variance of samples approximate the 
position 4 and the deviation o of the set of samples, respectively. 


n 


¢ Expected value of samples: ¥ = i as 


i=1 
; , 1 n sed 
e Variance: s° = —_ -x)y. 
n-14 
i=1 
In the case of a Gaussian RV, the joint distribution of the expected value and variance of 


the samples is defined as: 


1. X~N(pU,07/n). 


3. x and s’ are statistically independent RVs. 


The latter results reflect the fact that the normalized ratio of the expected value and vari- 
ance of samples is Student's t-distributed: 


Proof of properties (2) and (3): 
First of all, we will show that the expected value and variance of samples are indepen- 
dent RVs. Let the vector form of RVs y = [y,,--, y,]7, which is defined as follows: 


Ui Vinx = hx, 
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where 


ie Aan 


Note that the vector h, has a unitary norm. The Gramm-Schmidt orthonormalization 
method (Gramm-Schmidt process) is then applied to the vector h,. According to this 
method, n — 1 vectors h,,---,h,, are created, which are orthonormal, orthogonal to each other 
and with respect to h,. The vector form of RVs y is defined as: 


y =H'x, 


where H = [h,,---, h,] depicts a 1 x n orthogonal matrix. Because x = Hy, the orthogonality 
of H introduces the following properties: 


a. y~N,(0,D) 
b. yly = x'x. 


Since y, = Jn X, considering (a), it is deduced that value X is independent of the values 
YyY,- Moreover, using the equivalence of terms: 


n n 
S@ =z) = Sa “HX, 
i=1 1 


(b) and the RV y, definition yields: 


n n 


Voir = Vy vis yt ty. (78) 


i=1 i=1 


Hence, the variance of the samples is a function of y,,--:, y,, and is therefore independent 
of the expected value y, of the samples. Also, since y;,---, y,, are independent RVs M(0, 1), 
Relation 78 expresses the fact that the normalized variance of the samples follows a chi- 
squared distribution with n — 1 degrees of freedom. 

Property (3) is derived directly through the Fisher-Cochran theorem. The normalized 
variance of the samples on the left part of Relation 7.8 can also be expressed in a squared 
form: 


[x—1x]'[x-—1x]=x" saa 11a" af 
nN nN 


idempotent 
=x? [rar 1x, 
n 


orthogonal projection 


where 1 = [1, ---, 1]”. 
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NOTE: Since rank| Ia 1] =1-n, then: 
n 


[x—1x]'[x-1x]=(n-1)s’ ~ 7,4. 


7.5 Statistical Sufficiency 


Many detection/evaluation/classification problems have the following common structure. 
A continuous time waveform {x(f):  € R} is measured in n time-frames f,,---, t,, creating 
the following vector: 


x =(eoel ; 


where x; = x(t). Vector x is modeled as a vector of RVs with a joint distribution, which is 
known, but is dependent on p unknown parameters 0 = [0,,+0,]". The following notations 
are true: 


e Xis the sample space derived from the random vector x. 

e Bis the event space of x, that is, the Borel subset in the R” field. 
e 0 € Ois the vector of unknown parameters of interest. 

e @is the available parameter space of the current measurement. 
e Pyis the probability of measurement in B for conditional 0. 

e {Po}oco is defined as the statistical model of the measurement. 


This probabilistic model produces the joint CDF associated with x: 
F,(x;0) a Pr[x, SX My S Xn], 


which is known for each 8 € ©. When x is a vector with continuous RVs, the joint CDF is 
determined by the corresponding joint PDF, which will then be listed next in different 
ways depending on the content, f(x), f(x; 8), or f(x; 8). 

Moreover, the statistical expected value of a RV z is defined, in relation to the PDF, as: 


BalzI= | z f.(2i0)dz. 


The class of functions {f(x; 9)} .<,ec@ determines the statistical model of the measurement. 
The overall objective of the statistical inference can now be stated: Taking into account 
a vector x, we can extract the properties of ®@ knowing only the parametric form of the sta- 
tistical model. Therefore, we wish to use a function that maps the values of x to subsets of the 
parametric space, e.g., a suitable estimator, classifier or detector of 0. As we will see next, 
there are several ways to form inference function, but the fundamental question is: Are there 
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any general properties which such a function must have? One important property is that 
the statistical function must depend only on the x vector of length n, through a vector of 
smaller length, called sufficient statistical metric. 


7.5.1 Statistical Sufficiency and Reduction Ratio 


Let us consider as a statistical measurement any function of the given data T = T(x). There 
is a remarkable interpretation of a statistical feature regarding memory storage require- 
ments. Consider a special computing system that records the time samples of the vector 
x=[x,,---,X,]', x, =x(t), e-g., ina “byte” of the storage memory as well as the time tag t, 
in another “byte” of the memory. Any non-invertible function T, which maps the data 
domain from R” to R”, represents a dimension reduction of the data samples. We can 
evaluate the rate of reduction of the T-function by introducing the concept of the reduc- 
tion ratio (RR): 


R= # bytes of storage required for T(x) 
# bytes of storage required for x ~ 


This ratio is a measure of the data compression caused by a particular transformation 
through T. The number of bytes required to store vector x with the corresponding time 
tags is: 


# bytes{x} = # bytes{x,,---,X,, \"=# bytes{timestamps} + # bytes{values} 
=2n. 


Next, some examples are presented. 

Let x, be the i'* largest (sorted) value of x so that xq) > X) >> Xq). These values are 
called sorted statistics values and do not include time tag information. The Table 7.1 below 
illustrates the reduction ratio (RR) for some important cases. 

A discussion could arise about the maximum RR value applied so that no information 
about the parametric vector @ is lost. The answer is given by a specific metric known as the 
minimal sufficient statistic. 


TABLE 7.1 

Reduction Ratio (RR) for Some Cases 
Statistical Data RR 
T(x) = [egy cise HF 1 
T(X) = [Xay, «++ Xan]” Y 
T(x)=% 1/(2n) 


T(x)=[z,8?]}- 1/n 
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7.5.2 Definition of Sufficient Condition 


The metric T = T(x) is sufficient for the parameter 0 if it contains all the necessary informa- 
tion regarding the data samples. Therefore, when the metric T has been calculated, it can 
be stored for further processing without the need for storing the actual data samples. 

The metric T = T(x) is sufficient for the parameter 0 if it contains all the necessary infor- 
mation regarding the data samples. Therefore, when the metric T has been calculated, it 
can be stored for further processing without the need for storing the actual data samples. 

More specifically, let x have a CDF F,(x, @), @-dependent. The T statistic is sufficient for 
vector 0 if the conditional CDF of x, which is T = t-dependent, is not a function of 0: 


Fur(x|T =t,0)=G(x,t), (79) 


where G is a 8-independent function. 
With respect to the discrete values of the vector x with a probability mass function p(x) = 
P,[x = x], T = T(x) is sufficient for the 0 if: 


P,[x =x|T =t]=G(x,?). (7.10) 


For continuous values of x with PDF f(x; 0), Equation 79 is transformed in 
far (x | t, 8) = G(x, £). (711) 


In some cases, the only sufficient statistical quantity is a vector, eg. T(x) =[T,(x),---, TQ]. 
Then, it is true that T, are jointly statistic sufficient for 0. 

Equation 7.9 is often difficult to implement because it involves extracting the conditional 
distribution of x dependent on T. When the RV x is continuous or discrete, a simple way to 
investigate the sufficient condition is through the Fisher factorization property. 

Fisher factorization: T = T(x) is a sufficient statistical quantity if the PDF f(x; @) of x can be 
expressed as: 


£.(x;8) = g(T,8) h(x), (7.12) 


for some non-negative functions g and h. Fisher factorization can be used as the functional 
definition of a sufficient statistic T element. A significant effect of Fisher’s factorization fol- 
lows: when the PDF of the vector sample x satisfies Equation 7.12, then the PDF f;(t; 0) of 
the sufficient statistic T is equal to g(t, 8) with a range of values of up to a 0-independent 
constant q(t): 


fr(t;®) = g(t,8) q(b). 
Examples of sufficiency follow: 


Example 1: Use the Entire Symbolic Sequence of Data Samples 


Vector x = [x,,---, x," is clearly sufficient, but it does not constitute an efficient method. 
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Example 2: Use of Sample Classification 


Vector [Xq), ++, Xq] is sufficient when x; are IID. 
Proof: 
If x; are IID, then the joint PDF is expressed as follows: 


fol(X10+%,) = [Lac = [Lae 


The sufficiency is obtained by Fisher factorization. 


Example 3: Binary Likelihood Ratios 


Let @ take only two values, 0, and 0,, for example, a bit with a value range “0” or “1” at 
an information transmission channel. Then, the PDF f(x; @) can be expressed as f(x; 99) or 
fx; 9,), with a range of parameters @ € © = {0, 1}. Therefore, the problem of a binary decision 
arises: “Decide between 6 = 0 and @ = 1.” Thus, if there is a finite number for all values of 
x, the likelihood ratio A(x) = f,(x)/fo(x) is sufficient for 8, where f,(x) = f(x1) and f,(x) = fx:0). 


Proof: 
We express f(x) as a function of 0, fo, f; and then determine the likelihood ratio A and 
apply the Fisher factorization method: 


fol) = Of, (x) + (1-8) fo(x) 
= Ge +(1= 9) f(x). 


g(T 8) N(x) 


Therefore, in order to distinguish between the two states @, and 0, of the parametric 
vector 9, we can exclude the individual data and take into account only the scalar suf- 
ficient statistic T = A(x). 


Example 4: Discrete Likelihood Ratios 
Let © = [0,,---,0,] and the following vector of p — 1 probability ratios 


fo, (x) 7 fo, ~ 
= [ A100), “y Aad], 


re=| I OF Foy . 


be finite for all x values. The specific vector is sufficient for @ parameter. An alternative 
way to determine this vector is by using the sequence {A,()}gco = Ay(x),-A,,4(0), called 
likelihood trajectory, around 6. 


Proof: 

We define vector p — 1 of the data u, = e, when 0 = 0, k = 1,---, p — 1 where e, = [0,---, 0, 
1, 0,---, OJ" is the column vector of the (p - 1) x (p - 1) identity matrix. Consequently, for 
all 6 € ©, the joint PDF is given by: 


fao= usl fo, (x), 


Or fee ea 


which satisfies the sufficiency according to Fisher factorization. 
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Example 5: Likelihood Ratio Trajectory 


In the case when © is the sum of the single-variant parameters 6, the likelihood ratio 
trajectory 


A(x) = | i le <0, (7:13) 


is sufficient for @. In the above relation, 6) denotes any reference point in © for which the 
trajectory is finite for the vector x. When @ is not single-variant, Relation 7.13 is trans- 
formed to a likelihood ratio surface, which is also statistical sufficient. 


7.5.3 Minimal Sufficiency 


What is the maximum reduction that can be achieved in a set of data samples without 
losing information with regard to the dependence of the model on 0? The answer to this 
question lies in the concept of minimal sufficiency. Such data cannot be reduced without 
loss of information. Alternatively, all other sufficient quantity data may be reduced to the 
value determined by the minimal sufficiency without loss of essential information. 
Definition: T,»i, eXpresses minimal sufficiency if it can be calculated from any sufficient 
statistic quantity T by applying a suitable transformation to T. In other words, if Tis a suf- 
ficient statistic quantity, there is a function q so that T,,;, = q(T) is true. 

Minimal sufficient statistics are not unique: if Ti, expresses minimal sufficiency, then 
h(Tnin) Also expresses minimal sufficiency in case /t is an invertible function. The minimal 
sufficient statistics can be derived by many ways. One way is by finding a complete, suf- 
ficient statistic quantity. A sufficient statistic T is complete if 


Hele(T)]=0, VEO, 


which implies that g function is uniformly zero (g(f) = 0, V2). 

To prove that completeness involves minimal sufficiency, we quote the following discus- 
sion. Suppose M represents minimal sufficiency and C complete sufficiency. Since M is 
minimal, then it is a function of C. Therefore, ¢(C) = C —E,[C | M] isa function of C, because 
the conditional expected value E,[C| M] is a function of M. Since E,[g(C)]=0, V6,C is 
statistically complete, then C=E,[C|M], V0. Therefore, C is minimal since it is a func- 
tion of M, which is also expressed as a function of any sufficient statistic. Therefore, C 
‘inherits’ the property of being minimal from M. 

Another way to yield minimal sufficiency is by the reduction of data on the likelihood 
ratio surface. As in Example 5, 0) € © a reference point so that the following ratio reduction 
function is finite for each x € y and 6 € @: 


nals) = Ay) 


For a conditional x, consider a set of reduction ratios: 


A(x) = {No (%) focor 
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Definition 1: A (6-independent) function of x, t = r(x), adjusts the likelihood ratios A when 
the following conditions apply concurrently: 


1. A(x) = AQ, that is, A depends only on x through ¢ = ¢ (x). 
2. A(x) = A(7’), ie, t= 7, So the mapping t—A(z) is invertible. 


Condition 1 specifies another way of expressing the fact that c (x) is a sufficient statistic 
for 0. 


Theorem: If c = z (x) readjusts the likelihood ratios A(x), then T,,;, = 7 (x) expresses minimal 
sufficiency for 0. 

Proof: The proof concerns the case of a continuous x. The same result, however, 
also applies to discrete values of x. First of all, Condition 1, in Definition 1, implies 
that ct = 2(x) is a sufficient quantity. This is because, according to Fisher factorization, 
fo(x) = Ag(T) fo, (x) = 9(t;8)h(x). Next, let T be a sufficient statistic. Then, through Fisher 
factorization, f(x) = 9(T; ®)h(x), hence: 


ne=| fo(x) | -| g(T,®) : 
fo, (x) ae g(T, 8) <0 


Therefore, A(z) is a function of T. But, from Condition 2, the mapping t—A(z) is invertible, 
and ris a function of T. 

An equally important issue in practical applications is the size of the finite dimension of 
a sufficient statistic. 
Definition: A sufficient statistic T(x) is of finite dimension if its dimension is not a function 
of the number of data samples n. Often, the minimum sufficient statistics are finite. 


Example 6: Minimal Sufficiency of the Expected Value of the Gaussian Distribution 
Let x ~ N(0, 6”) with known o*. Find a minimal sufficient statistic for 0 = yw for the IID 
samples x =[x,,---,x,]'- 


SOLUTION: 
The joint PDF is given by: 


1 \ eoru? 
x)= eo 
fo(x) ae 
n eee ‘ 29 a 2 
een 
=- e i=1 i=1 
a 
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Consequently, using Fisher factorization: 


i=1 


which represents a sufficient statistic for wy. Moreover, since q(T) = 1'T is a 1-1 function 
of T, then x presents, equally, statistical sufficiency. 

It is proved, next, that the expected sample value is minimal sufficient, and depicts 
that it indexes the likelihood ratio trajectory A(x) = {Ag(x)}geq with @ = 1, © = R. The refer- 
ence point 6) = 9 = 0 is chosen such that: 


ful) uy we 
as fac men dL 20° } 


By stating T= »y x;, condition 1 of definition 1 is satisfied since A,,(x)= A ~s x;. Also, 
i=1 i 


according to Condition 2, the function ey is invertible to by for any non-zero 


value of 4. Consequently, the expected value of the samples indexes the trajectories, and 
expresses minimal sufficiency. 


Example 7: Minimal Sufficiency of the Expected Value 
and Variance of the Gaussian Distribution 


Let x ~ (0, 6?) with a known p and o?. Find a minimal sufficient statistic for 0 = [y, o7]" 
and for the ITD samples x = [X,,---,x,]". 


SOLUTION: 
n _ ~ x yp 
fo(X)= e eR : 
2207 
oN 5 x?- ; xptny 
: 1 : {3s run ’ 
210" 
T(x) 
iT 
i sgt #2 [ $5 
_ 1 - we 7a po 2 1 ; 
210" h(x) 
g(T,9) 
It is true that 


T 
n n 
T= 2 
(x)= Xiy Xi |, 
i= i=l 
—_en 


qT, Th 


is a (jointly) sufficient statistic for p,o?. Also, since q(T)= [n"T,,(n -1)°(,- T)| isal-1 
function of T, then s =[x,o7] expresses sufficiency. 
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As in Example 6, the minimal sufficiency of this statistical quantity results from the 
trajectory of the likelihood ratio {Ag(x)}geq With @ = [, 07], © = R x R*. Choosing arbi- 


trarily a reference point @, = [ Ly, 03 | =[0,1], we obtain: 


Ag(x) = £2) -(% ) dele [sda] 


tO oO 


where 6 = 


” Setting T= by Ds | Condition 1 is satisfied again. Condition 


2 needs further aes Although Ag(t) i is no longer an invertible function to t for each 
value of @ = [, 07], we can obtain two values 0 € {0,, 8,} in ©, for which the function of 
vectors Es (1), Ng, (2) | is invertible to t. Since this vector is defined by A(t), vector t 
‘follows’ the likelihood ratios. 

In order to formulate the invertibility relation, let 4 = Pu, Ay]? represent a pair of 


observed samples, resulting from the vector [ Aco, (tT), Ag, wy at A(t) surface. Now the 


problem is to define t from equation 1 = [ Ae, (1), No, (*)] . By talking the logarithm of 
each side of the equation, we find that it can equally be expressed as a 2 x 2 linear sys- 
tem of equations of the form 1’ = At, where A is a matrix of the elements @,, 0,, 8, and 4’ 
is a linear function of In 4. As a typical example, we verify the above if we choose 0, = 
[0, 1], 8, = [1, 1], 8, = [0, 1/2]. Then we have, 6 = 0 or 1 for @ = 9, or 8,, respectively, and A = 
diag(1, -1/2) is an invertible matrix. Therefore, we conclude that the vector [expected 
value, variance] of samples indexes the trajectories, thus expresses minimal sufficiency. 


Example 8: Minimal Sufficiency of the Position 
Parameters of The Cauchy Distribution 


1 
| 1+(x-6) | 


Let x; ~ f(x;0)= and x=[x,,---,x,]’ be the IID samples. 


It is true that: 


1 1 


F(x;8)= I] ; 
civ -or] = [Te _or 


We can see that the denominator consists of a polynomial of degree of 2 at 8, whose 
coefficients cannot be determined without determining the set of all possible cross 
products x;,,--- Xi, p= 1,2,---,n, of x; Since this requires the whole set of sample values 
to be determined: "there i is no » finite dimensional sufficient statistic. However, each of the 
individual cross product i is independent of the ordering sequence, and thus, the ordered 
statistic [Xq),--",X~y]' expresses minimal sufficiency. 


7.5.4 Exponential Distributions Category 


Let vector @= [0,,---,8,]" receive values in the parametric space ©. The PDF f, of the RV x is 
a member of the category of the exponential p-parametric distributions if for all ® € © it is 


fox) =a(O)b(x)e"", coc x <+o0, (7.14) 
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for some scalar functions a, b and p-parameter vector functions c and t. A similar definition 
applies to vector RVs x. 

It is noteworthy that for each fy of the exponential distribution categories, the supported 
set of values {x : fy (x) > 0} does not depend on the 0. Also, the same length of the vector 
c(®) = t(x) = p must be respected. This ensures that the size of sufficiency is of the same 
dimension with the parametric vector 0. 

Parametrization of the exponential distribution category is not unique, i.e., these distri- 
butions are not affected by any changes in parametrization. For example, if f,, 0 > 0 belongs 
to the category of exponential distributions, then if a = 1/0 and g, = f,,, it is true that 
8, 4 > 0 also belongs to the category of exponential distributions, possibly with different 
functions a(.), b(-), c() and t(). 

As a typical example, let the redefinition of the parameters use the mapping c: 8 > n and 
let c(0) = n be a reversible function. Then, using Relation 7.14, fy can be expressed as: 


fix(X) = A(M)b(x)e™"™, co < x <t00, (715) 


with a(n) = a(c(y)). Therefore, f, remains in the category of exponential distributions. 
When it is depicted as in Relation 7.15, it is expressed in normal form with normal param- 
eterization n. In this case, the expected value and covariance of the sufficient statistic 
T = t(x) results as: 


E,[T]= V Ina(n), 
and 
cove[T] = V7 Ina(n). 


Another useful parameterization of the exponential distribution class is the parameter- 
ization of the expected value. In this case, the functions a(-), b(), c() and t() of Equation 7.14 
are manipulated in an appropriate way so that: 


EelT]=0. (7:16) 


As we will note in detail in the next chapter, when a category of exponential distribu- 
tions is expressed through the parameterization of the expected value, the sufficiency T, is 
a minimal variance estimator of 0. Thus, the parameterizations of the expected value are 
absolutely advantageous. 

Examples of distributions belonging to the exponential category include: the Gaussian 
distribution with unknown predicted value or variance, the Poisson distribution with 
unknown expected value, the exponential distribution with unknown expected value, the 
Gamma distribution, the Bernoulli distribution with unknown probability of success, the 
binomial distribution with unknown probability of success, and the polynomial distribu- 
tion with unknown cell probabilities. 

Distributions not belonging to the exponential category are: the Cauchy distribution 
with unknown expected value, the uniform distribution with unknown support, the 
Fisher-F distribution with an unknown degree of freedom. 


404 Digital and Statistical Signal Processing 


When a statistical model belongs to this category, the sufficient statistics, for the model 
parameters, have a simple yet specific form: 


n 


folx)= [[e@rene 


i=1 
T 


c™ (6) t(x;) 2 


=a'@e | Joa). 
¢(T,0) i=l 


h(x) 


For this reason, the following vector, of p-dimension, is sufficient for 0: 


Ye = Ser Lyo ; 


In particular, the above vector is of finite dimensional sufficient statistic, which is com- 
plete and minimal. 


7.5.5 Checking Whether a PDF Belongs to the Exponential Distribution Category 


Due to the many attractive properties of the exponential distribution category, in many 
cases, the first question to be answered becomes: Is the PDF of the data x a member of 
this particular case? This question can arise, for example, if the input to a known filter 
or another system has a known PDF, and one can calculate a mathematical representa- 
tion of the PDF of the filter output. To check if the output PDF is exponential, it must be 
transformed in an exponential form. If this is difficult, the next step is to try and show 
that density does not belong to that specific category. Certain properties can be checked 
immediately, e.g., that the parametric space, ©, does not depend on the x range (as in the 
case of the uniform PDF with an unknown support region of the distribution boundaries). 
Another way is by computing 0?/d00x In f,(x) and verifying that it cannot be expressed in 
the “discrete” form c’(6)t’(x) for some functions c and t. 
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Fundamental Principles of Parametric Estimation 


In Chapter 7, we investigated the basic principles of statistical evaluation: the formulation 
of a statistical model and the statistical sufficiency for the model parameters. In this chap- 
ter, we continue with the analysis and derivation of methods for estimating the parameters 
from random samples of the model, paying particular attention to how accurate these 
estimates remain in different sampling realizations. 

We shall begin with the basic mathematical formulation of estimation, and then, specify- 
ing for the case of scalar parameters, we will study two different cases: the random param- 
eters and the non-random parameters. For random parameters, one can estimate the mean 
accuracy of the estimator easier and define procedures for deriving optimal estimators, 
called Bayes estimators, providing the highest possible accuracy. More specifically, three 
different optimization criteria are defined: the mean square error (MSE), mean absolute 
error (MAE) and mean uniform error, also called high error probability (P,). Afterwards, 
the deterministic scalar parameters are studied focusing on how unbiased the estimator 
is and on variance, which measures the estimation accuracy. This leads to the concept of 
Fisher information and Cramer-Rao lower bound to the variance of unbiased estimators. 
Finally, we generalize this analysis to the case of multiple (vector) parameters. 


8.1 Estimation: Basic Components 


In this chapter, the following notations apply: 


e xey isa vector of observed RVs. 

¢ Xis the sample space of realization of the x vector. 

e 0¢Qisa vector of unknown parameters of interest. 

e ©CR? is the parametric space. 

f(x;®) is the PDF of x for conditional 6 (known function). 


The goal of parameter estimation is to design an estimator function 6 = @(x), which cor- 
relates y at R?D ©. This is illustrated in Figure 8.1. 

It is important to distinguish between an estimator, which is a function of sample x, and 
an estimate, which is an evaluation of the function in a particular realization of x, i.e.: 


e The function @ is an estimator. 


e The point @(x) is an estimate. 


A reasonable question arises: What is the appropriate design criterion for constructing 
an estimator? There are many possible approaches that give an answer to this. In this 
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xX 0, 


FIGURE 8.1 
An estimator of vector 8 with conditional sample vector of RV x is matched as y at R?. 


chapter, we will describe two basic approaches. The first one assumes that 0 is random, 
while the second assumes that it is predefined (known). Common in both approaches is 
the description of a loss function, also called a risk function, which is associated with an 
estimator that measures the estimation error as a function of both the sample value and 
the parameters. 

Let c @(x);0) be a risk function associated with @ for conditional @ and x. The optimal 
estimator, if any, can be derived by minimizing the mean loss, E[C], where the capital let- 
ter C denotes the RV c(@(x);8). 


Dc ! 


8.2 Estimation of Scalar Random Parameters 


When 9 is scalar, i.e., 8, we have direct access to the following: 


e (0): a priori PDF of 6. 
° f(x|@): conditional PDF of x at 0. 
¢ f(6|x): a posteriori PDF of 6 determined by the Bayes rule: 


_ fx|6) F(0) 


fOlxy= 


¢ f(x): marginal PDF, which results by marginalization over 0: 


flx)= | folofeoae. 
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C(6,0) C(6,0) C(6,0) 


6-0 6-0 6-0 


FIGURE 8.2 
Loss functions for estimating scalar parameters: (a) square error, (b) absolute error, (c) uniform error. 


Based on the above, the mean loss or Bayes risk function can be calculated as: 


1CI= | [ e(60w;8) f¢<10) f(@)axc. 


At this point, we are able to define the optimal estimator. A scalar estimator, 6, which 
minimizes the average loss, is called the Bayes estimator. Some loss functions for this esti- 
mation problem are listed as follows: 


A A 2 
e c(6;0)= 6- r) : Square error. 


e c(6; 0)= 6 = | : Absolute error. 
° c(8;6) = 1(6- | > e) : Uniform error. 
Figure 8.2 depicts these three cases. 


For each of these three loss functions, the mean loss and Bayes risk functions can be 
calculated: 


¢ MSE (mean square error) estimator: MSE(6) = if 6- a | 
e MAE (mean absolute error) estimator: MAE(6) =E 6 7 @||. 
e Error probability: P, (6) = Pr[ (6 - | > e|. 


The optimal estimators @ that minimize the above criteria are still to be found. 


8.2.1 Estimation of Mean Square Error (MSE) 


The MSE is the estimation criterion most widely applied and undoubtedly the one with 
the longest history. The minimum mean square error estimator is the conditional mean 
estimator (CME) defined as: 


6(x) = E[O | x] = meang.g { f(0 |}, 
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where 


meang.o { f(@|x)} = Jerelxae. 


The CME has an intuitive mechanical interpretation as the center of mass (first moment of 
inertia) of mass density f(6|x) (Figure 8.3). The CME represents the a posteriori mean value 
of the parameter when the corresponding data sample has been obtained. 

The CME ensures the orthogonality: the error of the Bayes estimator is orthogonal to any 
(linear or non-linear) data function. This condition is mathematically expressed as: 


s| (0 6(x)) g(x)* ]=0. 


for every function g of x, where g* implies a complex conjugate of g. 


Proof: 
The MSE is expressed as: 


{|6-0P]= | (6- [| x])-(0- 10x) | 
= E(|6—Ef6|x}' |+E||o-EI0| x" | 
-B[ 9(x)*(4—E19| x1) |-B[ g¢)(9-Blo|x1)*], 


where ¢(x)= 6-E[ @|x] is an exclusive function of x. 


f(8|x) 


Ocme=E[O | X=x] 


FIGURE 8.3 
The CME minimizes the MSE. 
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Step 1: Express the orthogonality condition. 


8| ¢(x)(@-E[6|x]) 


I 
= 
— 
— 

o9 
Fhe 
tad 
oC 
Piet 
ca) 
| 
is 
rr 
ca) 
rad 
i 
— 

* 
rad 
maar 


Step 2: i 8 x | minimizes the MSE. 


{6-0 F1=B| 6-Blo|x1 |+B| 6—B[e| x1 | 
>B|o— Eo |xI" |, 


where the equality in the last relation occurs if and only if 6=E[0 |x]. 
8.2.2 Estimation of Minimum Mean Absolute Error 
(7) 
For simplicity reasons, consider a scalar real value of @ as well as F(@|x)= J f(0'|x)do’, a 


continuous function of 6. The minimum mean absolute error estimator is the conditional 
median estimator (CmE) defined as: 


6(x) = E[O| x] = medianyeo { f(0|x)}, (8.1) 


where 


ets) | Felnao- i 
Z ie (8.2) 
= mi ls | roca forae= | foxlaree) a 


The median of a PDF separates the function into two parts of equal mass density (Figure 8.4). 

When F(6|x) is strictly increasing in ©, the term “min” in the definition of the median is 
not obligatory, but is needed in the case of areas of © in which f(0|x) is equal to zero. If f(@|x) 
is continuous at 6, the CmE satisfies the orthogonality: 


s[ sgn (0 - 600) a(x) |= 0, 


and therefore, for the estimation of the minimum MAE, the sign of the optimal error esti- 
mate is that which remains orthogonal to each function of the data sample. 
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f(8 |x) 


Ocme=E[8 | X=x] 


FIGURE 8.4 
The conditional median estimator minimizes the MAE. 


Proof: 
Let @,, be the median of f(6|x). Therefore, from the definition of the median, it is: 


| sen(0- 6,,) x | = Jsen(0-6,00) £06 |x)do 
2) 

= [ felna- f felnae 
0>6y, (x) <6), (x) 


=0. 


Step 1: Express the orthogonality: 


s[ sen(0 -4,,)¢(x) |= | [san(0-6n)1x] 200) 


=0 


Step 2: For arbitrary 6, it is: 


MAE(6)= j| See 
A 


A ]+ | sen(@- 6,,)A | 


=0 


+ B[ sen(a +A)— sgn(a) |(a +A) 


>[ sgn(a+A)-1 |(a+A)20 


> E[|6-8,, |]. 
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Useful property: 
lat+Al =|a| +sgn(a)A +[ sgn|a+ Al- sgn(a) |(a + A) 


8.2.3 Estimation of Mean Uniform Error (MUE) 


Unlike the MSE or the MAE, MUE considers only the errors that exceed a certain threshold 
é > 0, where this approach is uniform. For small values of ¢, the optimal estimator is the 
maximum a posteriori estimator (MAP) (Figure 8.5): 


6(x) =argmax,.9 { f(0 x)} 


0) f(@ 
=argmaxeco {! a | (8.3) 


= argmaxeco | f(x|6)f(6)}- 
It is noteworthy that the last equality of Equation 8.3 is most preferred because it does not 
require the marginal PDF f(x), which may be difficult to calculate. 
Proof: 
Let ¢ be a small positive value. The probability of the estimation error being greater than 
€ is expressed as: 


P.(6)=1-P(|e—Al<e) 
= i JP} os ps f(0|x)d0. 


(8 |x) 


a 
Owap 


FIGURE 8.5 
The MAP estimator minimizes P.,. 


412 


Digital and Statistical Signal Processing 


Now consider the inner integral (at @) of the above relation. This is an integral at @ inside 


a window, called 2e-length window, centered at 9. With regard to Figure 8.6, if the value 
é is sufficiently small, the integer is maximized by centering the 2e-length window to the 


value of 0 that maximizes the embedded function f(@|x). This value is, by default, the MAP 
estimator for the parameter @. 


After the detailed description of the aforementioned estimation criteria (and their 


respective optimal estimators), some general observations follow: 


ie 


N 


» 


Ol 


ion 


The CME may not exist for a discrete © because the median may not be properly 
defined. 

Only the CME requires the (often difficult) computation of the normalized func- 
tion f(x) for the derivation of f@|x) = f(x|@)/f(x). 

All three estimators depend on x only via the function f(0|x). 


When the MAP estimator is continuous, unimodal and symmetric then each of 
the above estimators is identical. This is illustrated in Figure 8.7. 


. If T = T(x) is a sufficient statistic, the a posteriori estimation depends on x only 


through the median of T. So, if f(x|9) = 9(T;@)h(x), according to Bayes, it is true that: 


f@|x)= F(x|@)f@)  _ (79) F(@) 
[ felorode | scrie)sceyae 


which is a function of x through T. Therefore, with regards to the operation of the 
optimal estimation, absolutely no information is lost by compressing x to a suf- 
ficient statistic. 


. The CME has the following linearity property: For each RV @, and 6;: 


[0, + 0, |x] = E[6, |x]+E[6, |x]. This property is not true for CmE and MAP 
estimators. 


f(8|x) 


FIGURE 8.6 
The a posteriori PDF, which is integrated in the 2e-length window. 
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£(8|x) 


Bone = Bene = Omap 


FIGURE 8.7 
Symmetric and continuous a posteriori PDF. 


8.2.4 Examples of Bayesian Estimation 
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In this section, four typical examples of statistical models will be studied and optimal 
estimators will be suggested based on various criteria. 


Estimation of the width of the uniform PDF. 
Estimation of a Gaussian signal. 

Estimation of the size of a Gaussian signal. 
Estimation of a binary signal with Gaussian noise. 


Example 9: Estimation of the Uniform PDF Amplitude 


Consider the following problem: A network terminal requires random time to connect 
to another terminal after sending a login request at the time point t = 0. You, the user, 
wish to schedule a transaction with a potential customer as soon as possible after send- 
ing the request. However, if your machine is not connected within the scheduled time, 
the customer will address someone else. Assuming that the connection delay is a ran- 
dom variable x that is uniformly distributed in the time interval [0, 6], you can assure 
your client that the delay will not exceed @. The problem is that you do not know @, so it 
must be estimated from previous experience, e.g., the order of delayed connections pre- 
viously observed x,,---,x,. Assuming an a priori distribution at 6, the optimal estimate 
can be obtained using the theory developed above. 
Let x,,---,x,, be the IID samples, where each sample has a conditional PDF: 


1 
f(x, |@)= g liv. ™)- 

Based on past experience, we consider that: 
f()=0e"%, O>0. 


Figure 8.8 illustrates these two PDFs. 
Next, we will express the CME, CmE and MAP estimators for 0. 
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f(x] 8) #10) 


(a) (b) 


FIGURE 8.8 
(a) A uniform PDF of the unknown amplitude 8, (b) a priori PDF at 0. 


Step 1: Find the a posteriori PDF f(@|x) = f(x|9)f(@)/f(x): 
1 
f(x] 0) f(8) = TI Flin] e*) 
i=l 


6 1 
e 
a 9g" [i 
i=1 


Txay 400] (9) 
-0 


e 
= grt lew), 


where x.) = max{x}. It can be noted that the function is monotonically decreasing for 6 > 0 
(because the log derivative is negative). 
Also: 


flx)= | Focl@) flea 
= q-nsi(X 1), 


where g,, is a monotonically decreasing function: 


i= J 6"e ° do. 


Repeatable formula: q_,,,,(x) = ite - 1-0) yn=0,-1,-2,... 
n\x 
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Step 2: Derive the functions of optimal estimator. 


A 


Our = X(1) 


6 ome = 4—nv2(X(1))/4-nai (Xa) 
A ai 1 
Dome = rn G 4_n+1 (xa) ) ‘ 


Note that only the MAP estimator is a simple function of x, while the other two 
require more difficult calculations of the integers q,, and/or of the inverse function i 
The a priori PDF f(0|x), using these estimators, is depicted in Figure 8.9. 


Example 10: Estimation of a Gaussian Signal 


A common assumption that arises in many signal extracting problems is that of a 
Gaussian distributed signal observed with additional Gaussian noise. For example, a 
radar system can transmit a pulse to a detector for locating potential targets in an area. 
If there is a strong reflective target, then it reflects some energy from the pulse back to 
the radar, resulting in a high-energy signal, called signal-return. 

The range of this signal may contain useful information about the identity of the tar- 
get. Radar signal-return evaluation is complicated by the presence of noise produced in 
the radar receiver (thermal noise) or by interference from other sources in the coverage 
area. Based on radar field tests, the a priori average and variance of the received signal 
and noise may be available. 

Let two jointly distributed Gaussian RVs: s, x with known expected values, variance 
and covariance: 


Es] =U, B(x] = by, 


var(s)=o2, var(x)=0?2, 


cov(s,X) = po.0,. 


f(8|x) 


Ges Dome Oar 


FIGURE 8.9 
The CME, CmE, MAP estimations for the amplitude parameter 8 based on the a priori PDF, as derived from 
Figure 8.8 (b). 
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s represents the signal and x the measurement. Of course, the specific form of the covari- 
ance function will depend on the structure of the receiver—for example, it is reduced to 
a simple function of o, and o, for the additive noise model. 

We will find the optimal estimator of s with a conditional x. As with the previous exam- 
ple, the process of finding the CME, CmE and MAP estimators is presented in two parts. 


Step 1: Find the a posteriori PDF. 
A fundamental fact for jointly distributed Gaussian RVs is that if one of the variables 
is conditional, the other variable is also a Gaussian, but with a different expected value 
and variance, equal to their respective conditional values (Figure 8.10). 

More specifically, the conditional PDF of s with a conditional x is a Gaussian with 
expected value, 


Hegx(X) = i[s | x] =H, +p 


o. 
: (x a Ls) 
(oy 


x 


and variance, 


02, =E|(s-E[s|x1)" |x |=(1-p2)0?, 


so that the conditional PDF is of the form: 


Frist 18) f(s) 


fox (s | x) = Fx) 


(s ~ Hge(X)) 


2 
210 - 2 O5x 


f(s |x) 


Us |x 


FIGURE 8.10 
The a posteriori f(s|x) is a Gaussian PDF when x, s are jointly Gaussian. 
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Step 2: Derive the optimal estimator. 
As the a posteriori PDF is continuous, symmetric and unimodal, the CME, CmE and 
MAP estimators are of the same form. By deriving the dependence of the estimator 8 
from the observed realization x, it is: 


8(x) = My, (x) = linear at x. 


An interesting special case, related to the aforementioned example of the 
radar, is the model of independent additive noise, where x = s + v. In this case, 
=o +07, p= 02/(6? + 02), and therefore: 


Consequently, it is true that: 


* Minimum MSE: E| (s—§)* ]=(1-p?)o?. 
* Minimum MAE: E| (s—8)* |= /(1- p?)o2 J2/z. 
° Minimum P,; P[|s—4>¢]= 1-ert(e / \2(1- p?)o? ). 


Example 11: Estimating the Size of a Gaussian Signal 


We are now slightly changing Example 10. What if the radar operator was only inter- 
ested in the energy of the received signal and not in its sign (phase)? Then, we would 
aim to estimate the amplitude or magnitude, |s|, instead of the magnitude and the 
phase of s. Of course, a quick estimation process would simply be to use the previous 
estimator § and its magnitude | to estimate the real |s|. But is this approach the ideal 
one? 

We define again the two jointly Gaussian RVs: s, x with expected values, variance and 
covariance: 


E[s]=u,, El[x]=p,, 
var(s)=o2, var(x)=0%, 


cov(s,X) = po,0,. 


Now, the question is to estimate the conditional RV y = |s| in x. 


NOTE: y, x RVs no longer represent a jointly Gaussian model. The first step is to calcu- 
late the a posteriori PDF f,),. Since we know f,,, from the previous example, the problem 
is essentially presented as a simple transformation of the variables into a known func- 
tion. Using the difference method (Figure 8.11) we express the following relation, which 


is valid for small A: 


fux¥ lDA= forty | At far(-y |X), y20, 
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FIGURE 8.11 
Schematic representation of the difference method for deriving the conditional PDF of the RV y = |s| with 
conditional x by the probability: Ply <y< y + A|x) ® f,(y|x) A,0< A<<1. 


or: 


(y= Hast) | on| (y+ Hast) Toa). (8.4) 


1 
Fixly |x) = ‘prot, o*| 20%, 


In contrast to Example 10, the specific a posteriori PDF, is no longer symmetric to y. 
Therefore, we expect that the CME, CmE and MAP estimators will be different. 

The CME is derived in a closed mathematical expression, integrating the function 
¥f,(y|x) in ye[0,+00), according to the Relation 8.4, above, as follows: 


Maye 29] 2 -12 (203,.) 
sx(X)| exf} ——= |+,/—o,, e ‘ 
Ma(2) Ges aor 


On the other hand, from the equation of the minimum absolute mean error 


Yome(x) = Ely | x] = 


+ j 
J f(y |x)dy = J fixl(y|x)dy, the CmE, 7 = cn, is derived by solving the following 
7 0 
equation: 


Y — Ma,(X) Y + Mg(X) 1 
erf t+erf =_. 
| o,,V2 4.2 2 
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Finally, since the function f,),(y|x) is concave and smooth in y, the MAP estimator, 
Y = Yap’ occurs at a stationary point at y: 


of (y| x) 
ga 214) 
oy 


Using Relation 8.4, we obtain: 


(7 Mae (X)) (7+ Hy) 
: 205, 20%, 
y(x) = Hae (X) 
9 y.x)) 7 (7+ uye(x)) 
20%, ee 20%, 


It can be confirmed that when p,),/o, 


s|x-> +00, the three estimators converge to a com- 
mon point: 


I(x) > |Mye(2). 


This happens because the a posteriori PDF is transformed to the Dirac function, with 
a point of concentration y = H,),(X) aS [1);/0,),+00. It is noteworthy that none of the 
aforementioned estimators of |s| are expressed as a function of 5 , where § the estima- 
tion of s by CME, CmE, MAP estimators, as derived in Example 10. This fact indicates 
that the estimation of random parameters is not invariant with respect to the functional 
operators. 


Example 12: Sign Estimation of a Gaussian Signal 


In the previous examples, optimal estimators were studied for deriving the magnitude 
of a Gaussian RV, based on Gaussian observations. This example examines the case 
where only the phase of a signal interests, e.g., when a radar evaluates the sign of a 
signal. 

Consider the following observation model: 


x=0+W, 


where w is a Gaussian variable with zero expected value and variance o”, which repre- 
sents the additive noise. Also, the RV @ is a binary value with an equal probability of the 
two discrete states: P@ = 1) = P (@=-1) = 1/2, © = {-1,]}. This model is consistent with the 
radar problem with an a priori expected value yz, and additive noise. 

Here, the a posteriori PDF is the mass probability function, since the signal @ has 
discrete values: 


F(x | 0)p(@) 


p(@| x)= fa)” 
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where p() = 1/2. For convenience, we have eliminated the indices regarding the follow- 
ing probability functions. Also, it is true that: 


: ; op (S001 


2n0 20° 


2 
1 op(O0") =. 
2no° 20 


F(x|@)= 


Hence: 


fl)= flx|@=1)5+ flel@=-)5. 


By following the steps below, it is evident that the MAP estimator constitutes a deci- 
sion rule with regard to the minimum distance so that the value @ would be chosen as 
the actual 0, which is the closest to the measured value of x: 


Oyap = ATZ MAX; f(x| A) 


=arg ming; 1 {@ - 6) } 
7 1, x20 
"| alee 

On the other hand, CME is expressed as: 


Ocme = (1)P(6 = 1| x) + (-1)P(@=-1| x) 
eT I.. _(x+IP 
_o9[-8 29" )-e9[-) 


~_f («=1) (e+e) 
op| cane } rex ae 


Based on the above examples, the following useful conclusions are drawn: 


= 


. Different error criteria reflect different optimal estimators. 

2. The estimators of optimal random parameters are not invariant regarding the 
functional operators. That is, if $(@) is an optimal estimator of (0), and @ is 
an optimal estimator of 6, it is true that: 


(0) # 9(6). 


3. When CmE and MAP exist, they receive values within the parametric space ©. 
The corresponding values of the CME may also be outside this area, e.g., if the 
values are discrete or if they do not belong to a convex set. , 

4. The stationary point condition (MAP equation) 0 f@|x)/d0 = 0 at @=Oy,p is 
useful only for continuous PDFs, which are differentiable and concave at con- 
tinuous values of # parameters. 
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ee 
8.3 Estimation of Random Vector Parameters 


We define a parametric vector 9¢@ CR’, @= [@ pets, | and anorm-s in the domain © as: 


p 1/s 
; ab ‘ : 
i=1 


Note that when s = +00, the norm is equal to the maximum value of |6,|. 
The estimation criterion a c(6, ¢) | described above should be generalized to contain 
vector parameters. Some characteristic generalizations of the above three criteria are: 


le 


e Estimator MSE: 


MSE@) =| \6- ol, |= S'e[6,-0,°] 


P 
i=1 


e Estimator MAE: 
Pp 


MAE(@) = [|| 6-6 |, ]= >» Bll 4, —6; I) 


i=1 


e Estimator of error probability —(0 < p < +os): 
p.6)=1-P(/6-l, <e). 


When p = +00, P, is the probability where the magnitude of at least one of the elements of 
the vector 0—@ exceeds the value e. 

The MAE criterion, also known as the final norm deviation, does not often offer unique 
optimal estimates for vector values. Below, we will focus on MSE and P,. 


8.3.1 Squared Vector Error 


P 
As the function MSE() =)’ MSE(6,) is additive, the minimum MSE vector estimator 


i=1 x 

achieves the minimum of each individual value MSE(6,), i=1,---, p. Therefore, it is deduced 
that this minimization is achieved by a vector that receives the individual scalar values of 
the CMEs for each element: 


BL 8, |x] 
(8, 1x] | 


Ocue = B[0|x |= 
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FIGURE 8.12 
The MAP estimate is located at the total peak of the function while the CME at the center of the probability mass. 


As in the case of the scalar estimation, the MSE estimator is the mass probability center of 
the multivariate a posteriori PDF (Figure 8.12). 


8.3.2 Uniform Vector Error 


For small values of ¢, similar to the scalar case, it is true that: 


Oyiar =arg maxe-0 f(8| x). 


8.4 Estimation of Non-Random (Constant) Parameters 


In order to calculate random parameters, we consider an a priori distribution, and we can 
define an error criterion for the total estimate, the Bayes risk criterion, which depends on 
the a priori information but not on a specific current parameter value. In the estimation of 
non-random parameters process, there is no a priori distribution. We can, of course, con- 
sider the problem of estimating non-random parameters as an estimate of random param- 
eters that depend on the value of the parameter, which we can call a real value. However, 
the derivation of the optimal non-random parameter estimation requires a completely dif- 
ferent approach. This is because if we do not have a priori distribution of the parameter, 
almost any reasonable estimation error criterion will be local, that is, it will depend on the 
actual parameter value. Therefore, we have to define weaker optimality properties than 
Bayes’ minimum risk, such as the unbiased criterion, according to which an efficient esti- 
mator of non-random parameters should work. 

As in previous sections, we shall first study the case of the scalar parameters @. At this 
point, it is obvious that we shall not use the conditional PDF symbol f(x|@) and, therefore, 
the formula f,(x) = f(x;@) will be employed. 
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MSEg 


8, 8 
FIGURE 8.13 


The MSE curve as a function of 0 for the simple estimator = @, of anon-random parameter. 


So, what could be the design criteria for scalar parameter estimators 6? An approach 
could be the minimization of the MSE: 


MSE, = E,| (6-6) |. 


Here, however, the following difficulty is encountered: If the actual value of @ is @), the 
constant estimator @=c approaches zero (Figure 8.13). 


8.4.1 Scalar Estimation Criteria for Non-Random Parameters 


Some possible criteria for designing efficient estimators are the minimax criteria. 


1. Minimization of the worst MSE case. Choose @ so that: 


max MSE, (6) = max Eg [(6 - 0)? |. 
2. Minimization of the worst case of probability estimation error: 
max F, = max Py 6-6 [> e]. 


If we are satisfied by minimizing an upper limit of the maximum P,, then we could quote 
Chebychev’s inequality: 


A y| (0-0) 
nl3—eppe}s Bel OO 


: (8.5) 


and work on minimizing the worst case of MSE. 
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Then, we give some weaker conditions that an efficient estimator must satisfy: being 
consistent and unbiased. 


Definition: 6, = A(x, x,) is (weakly) consistent if, for all 6 and all ¢ > 0, it is true that: 
lim P,(|@, —0|> €)=0. 


This means that 6, converges with regards to probability in the real value 0. Also, it 
implies that the PDF of the estimator is located around @ (Figure 8.14). In addition, from 
the Chebychev inequality of Equation 8.5, if the MSE tends to zero with n+, then 6,, is 
consistent. 

Usually, MSE can be calculated more easily than P,, while indicating that MSE converges 
to zero is a classic way for checking the consistency of an estimator. 

For an estimator, 9, we define the bias of the estimator at a point, 0: 


A 


b,(6) = E,[6]- 0. 


Similarly, the variance of the estimator is: 


var)(6) = Ey (6 z,41)° |. 


It makes sense to require an efficient estimator to be unbiased, that is, b,(8) =0V0cO. 
This suggests the following design approach: restricting the category of estimators so that 
they are unbiased and trying to find a suitable estimator that minimizes variance compared 
to the other estimators of the category. In some cases, such an approach leads to a very 
efficient, in fact optimal, unbiased estimator called uniform minimum variance unbiased 
(UMVU), as depicted in Figure 8.15. However, there are cases where being unbiased is not 


f(8; 8) 


FIGURE 8.14 


The function f (0;0) calculates concentration of 6 around the real value 0. 
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FIGURE 8.15 

The UMVU estimator is an unbiased estimator 6 that has less variance than any other unbiased estimator, 6. 
a desirable property for an estimator. For example, there are models for which there is no 
unbiased estimator of the parameter and others for which the biased estimator arises too 
high MSE. Fortunately, such models do not often occur in signal processing applications. 


Definition: 6 is a UMVU estimator in the case when for all 6 € © has the smallest variance 
compared to any other unbiased estimator @. Therefore, a UMVU estimator satisfies the 
relation: 


var,(0) < var,(@), 0€@. 


Unfortunately, the UMVU estimators rarely exist and only for a finite range of n samples 
X1,+X, Therefore, in most practical applications it is preferable to sacrifice the unbiased 
property in favor of designing efficient estimation procedures. For such estimators, there 
is an important relation between MSE, variance and bias: 


19[ (6-6)? |= io ((6- 1611) +( 61-6) | 


=, |(6—,{61) |+(#,(6|-6) +28,[6-B, [61 ]b,6) 
vary (6) bg(4) =0 


= var,() + b3(6). 


MSE, (8) 


The above relation implies that for a given MSE there will always be a tradeoff between 
bias and variance. 

Two important categories with regard to the design of efficient non-random parameter 
estimators are: 


e The method of statistical moments. 
e The Maximum-Likelihood Estimation method. 
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8.4.2 The Method of Statistical Moments for Scalar Estimators 


This method consists of deriving the parameter that achieves the best match between 
empirically calculated moments and real moments. More specifically, for a positive integer, 
k, let m, = m,(0) the k-order statistical moment: 


m, = ig [x*] = [x'¢(<i0) dx 


It is desirable to create a set of K moments so that a certain function, h, would exist such 
that the following condition is satisfied: 


6 = h(m,(0),---,,(0)). 


For example, consider that a closed-type expression 9(0) for the k-order moment E,[x*] 
can be computed and the function g(@) is invertible. Then, if only the value m, is known, 
without determining the value 0 for which it was calculated, we could recover 6 by apply- 
ing the inverse function: 


0=g"(m,). 


As function g"! recovers 6 from the moment of x, if we have access to IID samples x,,---,x, 
of f(x;6), then we can estimate 0 by using g"! at an estimated moment, such as the empirical 
mean value: 


which implies the following estimator: 
O= g(t). 


However, it is often difficult to extract a single moment that derives a corresponding invert- 
ible function of 6. Indeed, using only the k-order moment we can locate a restricting equa- 
tion that (0) =m, offers multiple solutions, 9. Inthese cases, other statistical moments are 
calculated to create more restricting equations aiming to a single solution. We will discuss 
more about this phenomenon in the examples that follow. 

Next, some asymptotic optimization properties are described for moment estimators. 
When the moments m, are smooth functions with respect to parameter 0 and the inverse 
function g" exists, then: 


¢ Moment estimators are asymptotically unbiased as n>+oo. 
e Moment estimators are consistent. 
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At this point, it is worth noting that moment estimators are not always unbiased for 
finite samples. There are, however, some difficulties that we sometimes encounter using 
this method, which are summarized below. 


1. Moment estimators are not unique. They are depending on the order of the 
moment used. 


2. Moment estimators are not applicable in cases where statistical moments do not 
exist (e.g., Cauchy PDF), or when they are not stable. 


An alternative approach that solves the above problems is to match empirical moments 
with real moments m, when k is a positive real value less than one. In this case, fractional 
moments may exist (when integer moments do not exist) and are very useful. 


Example 13: ITD RV Bernoulli Distributed 


Bernoulli type measurements occur when (binary) quantized versions of continuous 
RV are used, e.g., radar threshold signals (above or below a certain value), failure indi- 
cation data or digital media, e.g., measurements from the Internet. In these cases, the 
parameter of interest is usually the probability of success, i.e., the probability of the 
variable measured to be “logical 1.” 

The model is expressed by x =[x,,--,x,], to be IID and: 


x; ~ f(x;0)=6"(1-6)'*, x=0,1. 


It is true that ¢[0,1] or, more specifically, 6 = P(x; = 1), 1- 6 = P(x; = 0). 

Aim: Derivation of moment estimator for 0. 

For each k> 0B[ x* |= P(x; = 1) =@ so that all moments are the same and function g 
that matches the moments of 6 represents the identity map. Therefore, a simple moment 
estimator of 6 is the expected (mean) value of the samples: 


6=x. 


Obviously, 6 isan unbiased statistic, since Eg[x] =m, = 0. In addition, the variance gets 
a maximum value @ = 1/2 (Figure 8.16): 


var, (x) = (m, - m?)/n = 0(1-6)/n. 


Repeating, for this Bernoulli example, the order of the moment used in the moment 
matching process leads to identical ‘method-of-moments’ estimators. This behavior is 
very unusual. 


Example 14: IID RV Poisson Distributed 


Poisson measurements are found in a wide range of applications where sample val- 
ues are available. For example, in positron emission tomography (PET) the decay of 
an isotope at a particular position within a patient’s body produces a gamma ray 
that is recorded as a single “count” in a detector. The time recording of the periods in 
which these measurements occur and are recorded to the detector is a Poisson process. 
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FIGURE 8.16 
Variance of the moment estimator for Bernoulli RV. 


The total number of records over a finite time period is a Poisson RV with an input rate 
parameter determined by the mean concentration of the isotope. The aim of a PET sys- 
tem is the reconstruction, i.e., to estimate the distribution of rates over the imaging vol- 
ume. Poisson distribution is also often used as a model for the number of components 
or degrees of freedom that produce the measured values, e.g., the number of molecules 
in a mass spectroscopy measurement, the number of atoms in a molecule or the number 
of targets in a radar detection coverage area. 
Considering ITD measurement samples, the model is presented as: 


x 


x; ~ p(x;8) = oe, x=0,1,2,..., 
x: 


where 6 > 0 is the unknown rate. Also, it is true that m, = @. Therefore, as in the previous 
example, the moment estimator of 6 is the mean value of the samples: 


Alternatively, as relation m? = 0 + 0” is satisfied by the second moment, another moment 


n 


estimator is the (positive) value of 6, that satisfies the relation 6, + 63 = 1y7 ae 
that is: ae 
ae ~1+1+4y’ 
peas a 


2 
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As another example, m, can be expressed as m,=0+m; or 0=m,—m; =var,(x;). 
Therefore, the moment estimator is: 


n 


6; =x? 3%? =n S'(x,-3). 


i=1 


Among the aforementioned estimators, only the estimator of the mean sample value 
is unbiased for finite 1: 


E,[6,]=9, var,(6,) = 0/n, 


Ep(6,)]= 6,  var,(O,) = (26? +0)/n. 


Closed-form expressions for bias and variance of 6, do not exist. 
The following conclusions are summarized: 


1. 6, is unbiased for all n. 
2. 6, 05 are asymptotically unbiased for n--+00. 
3. The consistency of 6, and 6, is directly derived from the above expressions for 


the mean value and variance, as well as by using Chebychev inequality. 


8.4.3 Scalar Estimators for Maximum Likelihood 


Maximum likelihood (ML) is undoubtedly the most widespread principle of parametric 
estimation in signal processing. Unlike other methods, ML typically results in unique esti- 
mators that can be applied to almost all problems. 

For a given measurement, x, the likelihood function for 0 is defined as: 


L(8)= f(x;8), 
and the logarithmic likelihood function is defined as: 
(0) = In f (x;8). 
These expressions are functions of 6 for a constant value of x (Figure 8.17). 
The ML estimator, 6, is defined as the value of 6, which makes the data x “more prob- 


able,” ie, 6 determines the greatest probability for the x vector to be derived from the 
function f(x;@). Mathematically the following relations are derived: 


d= arg MAX geo f(x;0) 
= arg MaX¢.e L(A) 
=arg MaXg.¢ (8). 
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f(x;8) 


maxof(x;8) 


(x) (x2) 


FIGURE 8.17 
The likelihood function for 0. 


In practice, the ML estimate can be obtained by maximizing each monotonically increas- 


ing function L(@). 
Some important properties of ML estimators are: 


1. 
2: 
3. 


ML evaluators are asymptotically unbiased. 
ML estimators are consistent. 
Unlike the MAP and UMVU estimators, ML estimators are invariant at each trans- 


formation of the parameters: 


9 = 3(0)> O= 2(6). 


This is depicted in Figure 8.18 for monotonic transformations, but in fact it is gen- 
erally valid for each transformation. 


. ML estimators are UMVU asymptotically in the sense that: 


&.. 4 
ee) cay 
1 


where F, expresses a mathematical concept knownas Fisher information (described 
in detail below), and 1/F, identifies the fastest asymptotic rate of variance reduc- 
tion of any invariant estimator. 


. ML estimators are asymptotic Gaussian: 


Vn(6, -0) > z, 


where z ~ N(0, 1/F,()). In the above relation, the symbol (—) implies convergence 
of the distribution. This means that the CDF of RV Jn (@,-9) converges to the 
(standard Gaussian) CDF of RV z. 
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6 


FIGURE 8.18 
Invariance of ML estimator under a functional transformation g. 


6. The ML estimator is equivalent to MAP for a uniform a priori condition f() = c. 


7. If the ML estimator is unique, then it is expressed as a function of the data only 
through a sufficient statistic. 


Example 15: IID RV Bernoulli Distributed 


We can work in two ways to obtain the ML estimator: (1) by considering the whole 
observation sample x and (2) by considering only the sufficient statistic, T(x). 


First way: 
The likelihood function is the following product: 


L@) = f(x;0) = [Je a 6) ; 


It is desirable to rewrite it in the form: 


1 n 
n->, x 


ds 
L(@)=6" (1-6) © (8.6) 
= ori (1 = ey ; 
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Since this function is smooth and concave at 6, factorization at 6 yields the stationary 
point condition for the ML estimator, 6: 


(1-)x; - O(1— x) 


= 9 ryhy= F: 
OE ag e " (1-6) |roa 


Solving the equation (1- 6)x; = (1 = x,) =0 results in the ML estimator: 


6=x, (8.7) 


which is identical to the moment estimator analyzed previously. 


Second way: 
Using the Fisher factorization of Equation 7.12 to the PDF of Equation 8.6, it is true 


that T(x)= Die: is a sufficient statistic for 6. The distribution of T is binomial with 
i=1 
parameter 0: 


fitsn=| : Jra-om. t=0,--,n, 


where f; expresses the PDF of T. Defining t= nx reveals the fact that it is exactly of the 
same form (except of a fixed multiplication factor), as in Equation 8.6. Consequently, the 
ML equation remains the same as before, so that the ML estimator is identical to the one 
expressed by the Equation 8.7. 


Example 16: IID RVs Distributed by Poisson 


To find the ML estimator of rate of parameter 0, we express the PDF of the samples as: 


n g* : 
foc0)=[]—e°. 
ist “7° 


The likelihood function L@) = f(x;) must be maximized at 6 for deriving the ML esti- 
mator. It is convenient to analyze the logarithmic relation: 


A 


Ou, = ALY MAX, InL(6), 
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where it is: 


(0) = In f (x;4) 


=)" x, ne- n6- Inx,! 
k=l kel 


constant in@ 


=x In@-nO+c, 


and c is an independent coefficient. 

It can be easily verified (for example, through the second derivative) that the above 
logarithmic function is a smooth and strictly concave function of 0. Therefore, the ML 
estimate finds the unique solution 9 = @ from the equation: 


0= In f/30="—n. 


The above relation is the same as the solution derived for the first moment estimator: 


for which we already know that it is an unbiased estimate with variance equal to 0. 
Next, we study the asymptotic Gaussian property: 


ee-0=sh| 230-0) 


n 


==) (x,-4). 


Based on the central limit theorem, the above RV converges to a distribution with a 
Gaussian RV: 


B,| vn(z-6) |=0, 
var, (V/n(x-6)) =6. 


8.4.4 Cramer-Rao Bound (CRB) in the Estimation Variance 


433 


Cramer-Rao (Cramer-Rao bound - CRB) can be defined for random and non-random 
parameters. However, CRB is more useful for non-random parameters, as it can be used to 
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determine the optimal or almost optimal performance of a prospective unbiased estima- 

tor. Unlike the non-random case, for the random parameters the optimum estimator, and 

its MSE are functions of the known joint PDF of @ and x. Thus, there are more accurate 

alternative solutions than the CRB for approximating the MSE estimator, most of which 

apply the approximation of an integral that represents the relation of the minimum mean 

square error. Consequently, the case of the non-random parameters will be discussed next. 
Let 6 < © describe a non-random scalar value and: 


1. © describe an open subset in the R domain. 
2. f(x;0) describe a smooth and differentiable in @ function. 


What follows is the definition of CRB for a scalar, @. 
For each unbiased estimator @ of 6: 


var,(6) > 1/F(6), (8.8) 


where the equivalence occurs if and only if for some non-random value k,: 
p) P 
99 i f(%:8) = ke(O—-8). (8.9) 


Here, k, expresses a constant that depends on 6 and not on x. CRB is considered a very tight 
bound, and the Relation 8.9 reflects the so-called CRB tightness condition. 

The parameter F(@), in the CRB, is the Fisher information, which is expressed in two 
equivalent relations: 


F(6)= | (2 msrt0)) | 
=- | a inf) 


06" 


The latter can be used to indicate that the value k, in the tightness assurance condition (8.9) 
is actually equal to F(@). 

Before proceeding with some examples, consider a simpler version of CRB. There are 
three steps to express the CRB. The first step is to notice that the expected value of the 
derivative of the logarithmic function is equal to zero: 


_fafeo/ae 
Be| dln fo(x)/d0 |= Hy| CeO" 
[tn fotx¥/06] Fol) | 


) 
= |<, folxrax 


= 5 | foto 


=0. 
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The next step is to show that the correlation between the derivative of the logarithmic like- 
lihood function and the estimator is a constant: 


io (6¢x) - 8 16)) (Atos, folxV/09) |= [(6()- B16) % foxx 


) a 
=, | 80x) folxrax 
=E,[6]-0 
—. 1, 


where the result of the second equation of the previous step was used. Finally, we apply 


the Cauchy-Schwarz ( ) [uv] < E [wu] 5 [v"]) inequality so that: 


1= E3] (6(x)- B,[6])(dlog f,[xVa8) | 


< Bs (6¢x)- (4) || (A108 fle 0)'| 
= var,(6)F (6). 


Equality is true if and only if u = kv for some non-random constant, k. This result is consis- 
tent with the Relation 8.8 and gives the CRB. 


Example 17: The CRB for the Poisson Rate 
Let x = [x,,--+,x,], a vector with IID Poisson RV: 
oO -~6 


x; ~ f(x;0)=—e , x=0,1,2,... 
x! 


In order to find the CRB, we must first calculate the Fisher information. So it is: 


n 


In f(x;6)= }" x, n6-n6— Yn x,! 
k=1 k=1 
constant in 6 


and factorizing twice: 


Sin fxs0/a0 = = a 
= (8.10) 
2 In f(x;0)/98 =-4 a. 
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n 


As | s| =n, the Fisher information, with conditional n samples, is expressed 
as k=1 


n 
F,(@)=—. 
()=" 


The CRB for each unbiased estimator of Poisson rate 0 yields: 


Valy (8) 2 


a|o 


We note the following: 


1. From Example 14 we know that the mean value of samples ¥ is an unbi- 
ased statistical quantity with variance var6(x)=6/n. This coincides with 
the CRB. 

2. We can understand if the impartial estimator achieves the CRB, alternatively. 
From the first equivalence of Relation 8.10, we note the tightness assurance 
condition of Relation 8.9: 


ain f(xi0)/00= 19" x n= T(x 6). (8.11) 


k=1 Yo 


Then, the variance of X can be expressed by calculating the CRB. This 
alternative way may be simpler than the direct estimation of the estimator 
variance. . 

3. The expected value on the right part of Relation 8.11 is zero because @ is unbi- 
ased. This implies that: 


Eo| dln f(x;0)/00 |= 0. 


This is because the derivative of the logarithmic function at 6 is an unbiased 
estimator at the zero when @ is the real parameter. This observation is general 
and applies to any smooth and differentiable PDF. 


GENERAL PROPERTIES OF THE SCALAR CRB 


First Property: 

The Fisher information is a metric with respect to the mean (negative) curvature of 
the logarithmic likelihood function In f(x; 0) around the real value of the 9 parameter 
(Figure 8.19). 


Second Property: 
Let F,,(@) the Fisher information for an IID RV vector, x = [,,---,x,]. It is true that: 


F,(0) = nF (8). 
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curvature 


FIGURE 8.19 
The curvature of the logarithmic likelihood function In f (x; @) around the real value of the parameter 0. 


Therefore, for smooth likelihood functions and unbiased estimators, the variance 
var,(9) cannot be attenuated faster than 1/n rate. 


Proof of the second property: 
Since x = [x,,--+,X,,] is IID, it is true that 


F(x;0) = [1s («:9), 


SO as: 


For unbiased estimators, the CRB determines an unreachable range of variance as a 
function of n (Figure 8.20). Efficient unbiased estimators 6 = @(x,,---x,) present a vari- 


ance ranging up to var,(@) = O(1/1n). 
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FIGURE 8.20 

The CRB determines a unreachable range of variance, which is located under the CRB curve, appearing as the 
non-shaded area. Efficient unbiased estimators, with constant parameters, exhibit a rate of variance attenuation 
Ln. 


Third Property: 

If 6 is unbiased and var,(6) approaches the CRB for all 6, @ is called an efficient esti- 
mator. Efficient estimators are always UMVU (note: the reverse is not true). In addition, 
if an estimator is asymptotically unbiased and its variance is attenuated at the optimum 
rate: 


lim b,(0)=0, lim nvarp(0) = 1/E,(6), 


n—->+0° 


where F, is the Fisher information given the sample x, @ is considered an asymptoti- 
cally efficient estimator. 

The exponential distribution family (or category) plays a major role in this efficiency. 
More specifically, if x is a sample of a PDF belonging to the exponential distribution 
family with a scalar parameter 6, then: 


0= E,lt(x)], (8.12) 


F(6) = 1/var,(t(x)), (8.13) 


where F(@) is the the Fisher information with conditional sample x. Therefore, if we have 


a vector of samples of ITD RVs from such a PDF, then 6= ey, t(x;) isan unbiased and 
efficient estimator of 60. i=l 
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Fourth Property: 
Efficient estimators for 6 exist only when the probabilistic models describing them 
belong to the exponential distribution family: 


f(x;0) = a(0)b(x)e OO", 


as well as when E,| t(x) |=8, ie, the PDF is subject to a parameterization of its 
expected value. 


Proof of fourth property: 

For simplicity (and without loss of generalization of property) we study the case of a 
single sample n = 1 and © = (—cv,+0o). Let us recall, at this point, that in order for the 
equality to be true at the CRB, the PDF must be expressed in the form of: 


0 


59 Sf (x/8) = ky(6-8). (8.14) 


For constant 4, we integrate the left part of Equation 8.14 in 6 € [6,,6’]: 
fa 
[Sm foseyar = In f(x;0’)—In f(x;6p). 
% 
On the other hand, integrating the right-hand part of Equation 8.14 yields: 


a (ce (oe 
Jkot6-0)a0-6 [k, ao | k,0d8. 
Q% 8% @% 

0) ae) 


By combining the above equations, it is obtained that: 


tx) 


fehacC eae’. 
a(@) b(x) 


The above properties are studied in more detail through the following examples. 


Example 18: Parameter Estimation for Exponential PDF 


A non-negative RV x has an exponential distribution with an expected value 6 if the 
PDF of x is of the form f(x;@) = 0"! exp(—x/6), where @ > 0. This exponential RV is often 
used in practice to model service time or waiting time in communications networks and 
other queuing systems. We can easily show that this PDF belongs to the exponential fam- 
ily of distributions substituting a(@) = 67',b(x) = Ij ,..)(x), c(@)=—-@" and f(x) = x. Since 
E,[x]=0, the PDF is subject to expected value parameterization, and therefore it can 
be concluded that the mean sample value X is an unbiased estimator of 6. Moreover, it 
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is an efficient estimator and therefore is UMVU when n IID samples x = [x,,---,x,,] are 
available. 


NOTE: From the above we cannot conclude that the statistical quantity 1/x is an 
efficient estimator of 1/0. 


Example 19: Let the Vector IID x with x; ~ (0,07) 


The “bell” curve of the Gaussian PDF is met in a variety of applications, making it a 
benchmark. The use of this model is usually justified by invoking the central limit theo- 
rem, describing measurements or measurement noise as the sum of many small com- 
ponents, e.g., random atomic collisions, light diffraction, and aggregation of repeated 
measurements. 

Our first objective will be to find the ML and CRB estimator for estimating the mean 
value 0 of the scalar Gaussian RV with a known variance, o?. As the Gaussian PDF with 
an unknown mean value belongs to the exponential family, we can follow the same 
approach as above to find effective estimators. But let’s follow an alternative process to 
verify this fact. 


or 


In f(x;6)=-F ino) ar DH a) +c, 


where c is a constant. Calculating the first derivative yields 


din f/30= ar Dy -6) 
cs (8.15) 
= (3-6). 


Go; 
ko 


Therefore, the CRB tightness condition is valid. It is also concluded that x; is the 
optimal estimator for the mean value of a Gaussian sample. 

By taking the first derivative of the logarithmic likelihood function with respect to 
@ and reversing it, we confirm what we already know about the variance of the mean 
value of the sample: 


vaty(X) = 1/F, (0) =07/n. 


The first equality occurs because we already know that x is an efficient estimator. 
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8.5 Estimation of Multiple Non-Random (Constant) Parameters 


In this section, we are studying the general problem of multiple unknown parameters. 
This problem is different from the previous case of multiple random parameters since 
there is no joint a priori PDF for further processing. First, we model all unknown param- 
eters in a vector: 


6=[6,,---,6,]', 


and we describe the problem as the derivation of a vector estimator @ for 6. 
The joint PDF for the measured vector x is defined as: 


F(%;6,,--+,0,) = f(x;8). 


Criteria for estimator performance evaluation: 
We define the biased vector estimator: 


b,(0) = E9[6]- 8, 


and the symmetric matrix of estimator covariance: 


cove(@)= Bg| (8 16))(6- 16)" | 


vare(8,) cove(8,,0,) te cove(0,,8,) 


cove(8, F 6.) vaty(0,) 


| cov(6,,,6;) oo ve var, (8, ) 


This matrix is often called a variance-covariance matrix. 

In many cases, only the diagonal entries of the covariance matrix of the estimator are a 
point of interest. However, as we will see below, the entire covariance matrix of the estima- 
tor is very useful for generalizing the scalar CRB case. 

Also, the estimator concentration is defined as: 


p,(|6-0| > e) = (0;6)d0 
ne 
= } f(x;0)dx. 
{xl6-el>e} 


The first step is the generalization of the CRB from the scalar case (of one parameter) to the 
vector case, called a CRB matrix. 
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8.5.1 Cramer-Rao (CR) Matrix Bound in the Covariance Matrix 
Let 0 € © be ap x 1 vector and: 


1. © be an open subset in R’. 
2. f(x;0) be smooth and differentiable at 0. 


3. The coefficients cove(@) and F(@) (to be subsequently specified) are non-singular 
matrices. 


The CRB matrix for vector parameters is defined below. For each unbiased estimator 0 
of 0: 


cov, (6) > F (6), (8.16) 


where equality is true if and only if the following relation is satisfied for a non-random 
matrix Kg: 


K,Vo In f(x;0) = 6-8. (8.17) 


If the tightness condition of Relation 8.17 is satisfied, the estimator @ is called an efficient 
vector estimator. In the CRB matrix of Relation 8.16, F@) is the Fisher information matrix, 
which takes one of two following formats: 


F(@) =| (Vp In f(x;6))(Voln f(x;6))’ | 
=-E| V3 In f(x;6) |. 


where the factorization operator is defined: 
v,<| 2,2], 
| 00,’ "00, |’ 


and the Hessian symmetric matrix operator: 


Py v Py 
00 «= 00,8, «8,8, 
v Py 


V5=| 00,0, 002 


ee 


30.8, me ca 59? 


p 
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The CRB matrix of Relation 8.16 has some other properties in relation to the scalar CRB. 
First Property: 


The inequality in the matrix bound must be interpreted in the sense of positive definite- 
ness. In particular, if A, B are p x p matrices: 


A2BeA-B20, 
where A - B 2 0 reflects that the matrix A — B is non-negative defined. This implies that: 
z'(A-B)z2=0, 


for each vector z € R? and all A — B eigenvalues are non-negative. For example, choosing 
z = [1,0,---,0]? and z = [11,---,1],, respectively, A 2 B, A > B implies that: 


a, 2b, and a; , 2 b,j. 
i,j i,j 


However, A 2 B does not mean that a; 2b; is generally true. A simple numerical example is 


the following. Suppose that 0 < p < 1 and: 


where two eigenvalues 1 — p > 0 and 1 + p > Oare present. Therefore, it is derived that A — B > 0, 
as 4,,=O0#p. 


Second Property: 
The inequality matrix of Relation 8.16 is also true for the scalar CRB that expresses the 
variance of the i-th element of the unbiased vector estimator 0: 


var,(6;) >[F7(@) | 


a? 


where the right part of the above inequality denotes the i-th element of the diagonal of the 
inverse Fisher information matrix. 


Third Property: 
The Fisher information matrix is a metric that expresses the mean curvature of the loga- 
rithmic likelihood function around 0. 
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Fourth Property: 
Let F,,(0) be the Fisher information for a vector sample with IID measurements %,,--+,,,. 
Then, it is true that: 
F,(0) = nF,(@). 
Thus: 
var,(@) = O(1/n). 


Fifth Property: 
Efficient vector estimators exist only for multivariate parameters belonging to the expo- 
nential distribution family with expected value parameterization: 


f(x;0) = a(8)b( xe to” Leo), 


and 


| t(x) |= 0. 


In addition, in this case, x) =6,and @= nS t(x;) is an unbiased and effi- 


cient estimator of 0. i=1 i=l 


Sixth Property: 
@ satisfies the relation: 


V, In f =K,(6-8), 
for a non-random matrix, Ky, we conclude that: 


1. @ is an unbiased statistical quantity as it is true that: 


| Vo ln f(x;0) |=0. 


2. @ is an efficient estimator, and therefore its elements are UMVU. 
3. The covariance @ is given by the inverse Fisher information matrix. 


4. The matrix Ky is the Fisher information F(®) since it is true that: 


dg [ V3 In f(x;0) |= Jo| V5Vo ln f(x;6) |= iol Vo {Ko(0—0)} |, 


and via the chain rule and the unbiased property of 6: 


ig| Vo{Ko(0—0)} |= VolKo}Ho| 6-0 | + Kgig | Vo(0~ 0) ]=—K,. 
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5. The covariance of the estimator is expressed as: 
cove(8) = Kg'. 


Proof of the CRB matrix: 

There are three steps to the proof, which, with one exception, is the direct generalization 
of proof of the scalar CRB: (1) deducing that the derivative of the logarithmic likelihood 
function is zero, (2) deducing that the correlation of the derivative of the logarithmic likeli- 
hood function and estimator is a constant, and (3) deducing that the derivative covariance 
matrix and estimator error implies a specific relation between the Fisher information and 
the estimator covariance. 


Step 1: Verification of Eg [Ve In f (x; 6) | =0. 


=i +f aay Yoh) |= J Yoon 


= Vo | f(xi8)dx =0. 


<a 
1 
A T 
Step 2: Verification of Eg lv, In f(x;0)(6— 9) iz 
We note that: 
Jo] Vo ln f(x;0)(6-6)' |=E y \7e5 A) Vol (x; 0)0 " 
= 1% F(x;0)0"x dx 
x 


aV 5 | 6:00" dx =I. 
a4 
EgL6? |-o" 


Therefore, combining the above result with that of the previous step, it is derived that: 


J9| Vo ln f(x;6)(6—8)’ | 
= E,| Vo ln f(x;6)6" |— Eg | Vain f(x;6) |6". 


=I 


Step 3: Let a 2p x 1 RV vector: 


7 6-0 
7 Voln f(x;0) } a 
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Since any matrix expressed as an outer product of two vectors is positively defined (i.e., 
ig uu’ |> 0), and using the results of the two previous steps, it is derived that: 


Jgfuu’ |= | — a | >0. 


We still have to apply the result of Appendix I to the above partitioned matrix so that: 
cove(8)—F (0) >0. 


We will then prove that the tightness criterion ensures the equality in the CRB. First of 
all, it is true that if cov,(®)=F™”, then the class of matrix E[uu’] is p. This can only hap- 


pen if the RV vector of Relation 8.18 has p independent linear coefficients. As cove (8) and 


F@) are considered non-singular parameters, vector 6-6 as well as V, In f has no linear 
correlations. 
Therefore, it is: 


K,V,ln f =6-8, 


for a non-random matrix. It follows that the derivative of the logarithmic likelihood func- 
tion is located in the error range of the estimator. 


8.5.2 Methods of Vector Estimation through Statistical Moments 


Let m, = m,() a moment of k-th order of f(x;0). The vector that expresses the process of 
estimating statistical moments involves K moments so that the vector function of 0 € R”: 


g(8) =| 1,(@),---,1,(8) |, 
can be reversed, i.e., a unique 9 exists that satisfies: 
0= g (mM, . “+, Mx). 


As in the scalar case, the moment estimator is constructed by substituting m, with its cor- 
responding empirical estimation: 


where 


Fundamental Principles of Parametric Estimation 447 


8.5.3 Maximum Likelihood Vector Estimation 


The ML vector is an obvious generalization of the corresponding scalar case: 
6= arg MAXgco f(x;9). 
For smooth likelihood functions, the ML vector has several important properties: 


1. The ML vector estimator is asymptotically unbiased. 
2. The ML vector estimator is consistent. 
3. The ML vector estimator is invariant to any transformation: 


9=g(0) => o= (6). 


4. The ML vector estimator is asymptotically efficient and, therefore, its individual 
elements (estimates) are UMVU asymptotic. 


5. The ML vector estimators is asymptotic Gaussian: 
Vn@-0)>z, z~N,(0,F,1()), 


where F,(9) is the Fisher information matrix of the sample: 


F,(0) = —E,| Vlog f(x,;6) |. 


Example 20: A Joint Estimation of the Expected Value and 
of the Variance of a Gaussian Sample 


Let a IID sample vector x = [x,,---,x,] that consists of a Gaussian RV x; ~ M(u, 07). The 
unknown parameters are 0 = [j,07]. 
The likelihood function is defined as: 


(8) = In f(x;@) =——In(o?) : (x,- p> +e. 8.19 
; 6.19) 


Step 1: Approach via the statistical moments method. 
It is known that m, =u, m, = 07 + p’, So: 


2 2 
=m, O =M,-—my,. 


Therefore, the moment estimator for @ is expressed as: 
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As usual, we define: 


where 


is the sample variance. 


Step 2: Approach via ML method. 


Since Relation 8.19 is a concave function (because —V% In f is positively defined), the likeli- 


hood equation (stationary point condition) can be used to find @=6@: 


ye -6,) 


0=Veln f(x;@)= ce 
50 DU 6.) 
Hence: 
Rahs, 6-62" 


and therefore, the statistical moment estimators and ML estimators are the same. 
Bias and covariance are easy to calculate: 


deli] =H, 


unbiased 

oan ae n-1 

alo] -( Jor, 
n 

Se a La 


biased 


Valg(X) = 07/n, 


vale) =("=") vane?) =( 22° }("™=1). 
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Since the expected value and sample variance are uncorrelated to each other (Appendix I): 


o7/n 0 


cov,(@) = : (20*\n—2) | (8.20) 


Then, we calculate the Fisher information using the expected value of the Hessian matrix 
—Vo In f(x;8) 


o*/n 0 
F(@)= P 8.21 
) | : aoe (6.21) 
by deriving the CRB: 
«| o/n 0 
COV» (8) | j Bo } (8.22) 


Some useful comments are to follow: 


1. Moments and ML estimators have covariance matrices that violate the CRB (this 
can be derived by comparing the data between the matrices of Relations 8.20 and 
8.22). This is expected because the ML estimator is biased. 


2. Let the (biased) corrected estimator for [j1,07]": 
7 T 
6=[x,s?]. 


— 


This estimator is unbiased. Since it is true that: s* = (Ye 
n 


We conclude that the covariance matrix of the corrected estimator no longer vio- 
lates the CRB. Indeed, it is an effective estimator of value jz since: 


vata (ft) =[F", = 0?/n. 
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However, s? is not an efficient estimator of o? because: 
2 ans 
Vala (s?)>[F ‘|. 


3. The estimator ML is asymptotic efficient when n — +00: 
A o 0 2 
NcoVv (0) = ‘(*=1) “l° 0 fre. 
Oe 20% 
n 


4. We can verify that the vector [f1,07] is asymptotic Gaussian simply by noting that: 


— 


¢ {,0° are independent RVs. 

© Vn(fi-u)~N (0,07). 

© Vin(s?— 02) ~ 0? Vn(y2,/(n-1)-1). 
e 4¥,~N(v,20), 0> +6. 


5. Based on the CRB equality condition, it is possible to recover an efficient vector 
estimator (but not for 9): 


Vo In f(x;0)=K ae 
X; = — 
: | x2-(0? +1") 


where 


-1 
— n/o 0 1 0 
: 0 n/(2o4) || 2H 14° 


Since the sampling moments are unbiased estimates, we conclude that x, x? are 
efficient estimates of the first order moment E[x]=f and of the second order 


a] x2]=02 + ”, respectively. 


Example 21: n =[N,,---,N,]" is a Polynomial Random Vector 


The polynomial model is a generalization of the binomial, with more than two output 
states “0” and “1.” Let the output vector z one of the elementary vectors belonging to R?, 
e= Roe leer = (Orties A |; with probabilities 0,,---,0,, respectively. The vector 
e, may be a tag related to the fact that, for example, a random roll of a dice resulted in the 
number 6 (p = 6). The polynomial model describes the distribution of the sum 


of the vectors obtained after n IID trials. 
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The probability of a particular polynomial result n has a mass probability function: 


n! N. N 
p(n;0)= —™! __9...6"?, 
N,! 


Na des 


p p 
where N; > 0 are integers that satisfy the equations SIN ,=Nn, y'4, =1,0, €[0,1]. 
i=1 i=1 
The moment estimator of @ is obtained by matching the first empirical moment n with 
the first statistical moment, E,[n]=0n. This has as a result the estimator 8 = n/n or: 


=| . a 
n’ 'n 


The ML estimator should then be modeled. The p parameters exist in a p — 1 subspace 
in R? domain due to the limitation of the total probability. In order to find the ML esti- 
mator, a re-parameterization of the problem can be used or, alternatively, the Lagrange 
multipliers technique. We adopt the second technique as follows. 

We form the equation by including the restrictive terms as follows: 


Pp 
K-10 700-1| S01, 


i=1 


where d is the Lagrange multiplier, to be defined below. 
Since the function J is smooth and concave, we set its derivative to zero to determine 


the ML estimator: 
P 
0=Vo]()=Vo| >, N;ln8, - 26, 
i=1 
Ny. Noy 
0; e 6, 
Thus, 


Finally, we determine ) making the estimator 0 to satisfy the condition: 


YN/rs1s.=SN,=0 
i=1 i=1 


From the solution of the equation, the ML estimator is obtained, which is the same as 
the moment estimator. 
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8.6 Handling of Nuisance Parameters 


In many cases only one parameter, 0, is of interest, while the other unknown parameters 
05," 9, are nuisance parameters that are not a point of interest. For example, in Example 20, 
with unknown values of both mean and variance, the variance may not be of direct inter- 
est. In this example, we found that the covariance of the estimator is a diagonal matrix, 
which means that there is no correlation between the estimation errors of the mean value 
and the variance parameter estimation errors. As we will see below, this means that vari- 
ance is a nuisance parameter, as knowledge or lack of knowledge of it does not affect 
the ML estimator variance for the mean value. We separate the analysis of the nuisance 
parameters in cases of random and non-random parameters. 


Handling of random nuisance parameters: 
For random values, the average loss affects the estimation errors only for 0: 


sl (6,,0,)]= 40, | axc(b,,0,) F016.) f(,). 


e; x 


The a priori PDF of 0, is calculated from the a priori PDF at 0: 
(0) = Jao, += [d6, f(0,,82,-,8,) 
The conditional PDF of x at 0 is derived: 
F(x|6,)= Jao, + d6, F(x |0;,02,-++,8,) f (820-18) 161+ 
resulting in the a posteriori PDF of @;: 
fO,|x)= Jae, += d6, FO, 20-8, |x). 


Note that the estimates of 0,,--+,8, are not necessary for the calculation of the a posteriori 
PDF of 0@,. However, integration (marginalization) of the conditional PDF in the range of 
the parameters O5,-+-/8, is required but may lead to significant difficulties in realization. 


Handling of non-random nuisance parameters: 
This case is quite different. The average loss concerns @,, again, but also depends on all 
the other parameters: 


e[C]= J-6.,6,) f(x |@)dx. 


x 
The ML estimator of 0, is expressed as: 


A 


6, =argmax, | max log f(x|6,,0,,---,0,)\. 
8 ee Oe 2 
ae, 
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It is worth noting that, in this case, the estimates of nuisance parameters are neces- 

sary because the aforementioned maximization process is realized by comparing all other 
parameters 82,--+/8y. 
CRB predictions for non-random nuisance parameters: 
As before, we consider only the case of the unbiased estimation of the first parameter 6, in 
the vector of the unknown parameters 0. The extraction of the CRB matrix in Relation 8.16 
made it possible for us to assume that there were unbiased estimators of all parameters. It 
can be proved that this limitation is unnecessary when we are interested only in 0). 

Let 0 = [6,, sul|” be the parameter vector of unknown values. The variance of an inde- 
pendent estimator 0, of 0; obeys the following rule: 


var,(6) >| F“(6) | (8.23) 


1? 


where the equality is true if and only if there is a non-random vector hy such that: 
hlV, In f(x;0) =(6, —9,). 


In Equation 8.23, it is true that: 


a0) ai@) — a6) 
00; 00,00, 06,00, 
a0) de) .. 
F(®)=-E]| 06,00, 003 ‘ 
0°1(0) 7 eee) 
| 96,00, 06, | 


and 1(8) = In f(x; 8). 
Let the partitioned Fisher information matrix: 


_|a b? 
ro-| 2 | 


a=—Ky [ 2 In f (x;)/067 | = The Fisher information for 0,, without nuisance parameters. 
__@P [ OV 5,8 In f(x;0) /06, | = The Fisher coupling of 0,, with nuisance parameters. 


= 49 


=—Ey hee In f (x ;0) | = The Fisher information for the nuisance parameters. 


Using the inverse property of the partitioned matrix (Appendix I), the right part of 
Relation 8.23 is expressed as: 


1 


LPO]. = ree 
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We can draw some useful conclusions from this result: 


° LF“) |, 21/a=1/ [ F(@) |,,. Consequently, the presence of nuisance parameters 
may reduce the performance of the estimator. 


e The magnitude of the performance reduction of the estimator is proportional to 
the amount of the information coupling between @, and 0,,---/0,,. 

e When the Fisher information matrix is diagonal, there is no reduction in the per- 
formance of the estimator. 


9 


Linear Estimation 


In Chapter 8, several approaches to the parameter estimation process were discussed, 
given the PDF f(x; @) of the measurements. All these approaches require accurate knowl- 
edge of the model, which may not always be available and/or feasible. In addition, even 
when the model is very reliable, these approaches usually yield estimators, which are 
non-linear functions of the measurements and whose implementation may be difficult; 
e.g., they may include the analytical maximization of a complex PDF or the analysis of its 
statistical moments as a function of 6. In this chapter, we present an alternative linear esti- 
mation approach that only requires knowledge of the first two moments or their empiri- 
cal estimates. Although linear methods do not have the desirable properties of the Bayes 
optimal estimators, such as MAP and CME, they are very attractive due to their simplicity 
and robustness to unknown variations in higher order moments. 

The theory of linear estimation begins with the assumption of random parameters by adopt- 
ing a square error loss function and then seeks to minimize the mean square error in all the 
processes of the estimator, which are defined as linear or affine functions of the considered 
measurements. It can be proven that the linear minimum mean square error (LMMSE) prob- 
lem can be redefined as the minimization of a norm in a linear vector space. This leads to an 
elegant geometric interpretation of the optimal LMMSE estimator via the projection theorem 
and the orthogonality principle of the norm minimization problems in the linear vector space. 
The resulting LMMSE estimator is dependent on the expected value and the variance of the 
measurement, the average of the parameter and the covariance of the measurements and the 
parameters. It is no surprise that when the measurements and parameters are jointly Gaussian 
distributed, the affine estimator is equivalent to the conditional optimal mean estimator. When 
the predicted value and variance are not a priori known, then a non-statistical linear least 
squares theory (LLS) can be developed a priori, leading to the problem of linear regression. 

As usual, the main components for the linear estimation will be the measurement vec- 
tor x= eel and the parametric vector 0=[6,,:- “2, ]'. The cases in which the latter 
vectors are RV realizations with known (ie., given) or unknown their first two statistical 
moments are studied in the following two sections. 


a 


9.1 Constant MSE Minimization, Linear and Affine Estimation 


Our objective is to derive the solution of the following minimization problem: 


A A 2 
min MSE(@) = min | |o— 6) | 


where the expected value is over both @ and x, as the minimization is limited to constant, 
linear, or affine functions @ of x. The norm in the above equation is the Euclidean norm 
|u| =.J/u7u. Then, the simple case of scalar parameters is discussed first. 
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9.1.1 Optimal Constant Estimator of a Scalar RV 


This is the simplest estimator because the constant estimator @=c does not depend on the 
measurements. It can be derived that the optimal constant estimator depends only on the 
mean value of the parameter as no additional information about the parameter distribu- 
tion is necessary. 

The problem is modeled as the derivation of the constant @=c that minimizes the MSE: 


MSE(c) = E[ (@—c)? ]. 


SOLUTION: . 
The optimal constant estimator is defined as 0 = E[6]. 
Since the MSE is a squared function of c, the above relation can easily be proved by set- 


ting the derivative < MSE() equal to zero. Alternatively, we can show that the following 
is true: ‘ 


Since only the second term of the last equation depends on c, which is a non-negative 
value, it is obvious that 6 = E[6] is the optimal constant estimator. Thus, 


min = MSE(c) = | (@- E[6)) |. 


which represents the a priori variance of 6, var(6). Since the constant estimator does not use 
information about the metrics, it is expected that any estimator, depending on the x-vector, 
will have a smaller MSE than variance var(6). 


9.2 Optimal Linear Estimator of a Scalar Random Variable 


The next estimator (which is a more complicated case) depends linearly on vector x: 
6=h'x, 


where h=[h1,---,h,]' is a set of linear coefficients to be defined. For the implementation of 
this estimator, the second moment matrix is necessary: 


M, =E[xx’], 


x 


Linear Estimation 457 


as well as the cross-moment vector: 


M,, 9 = E[x6]. 


Let M, be the invertible matrix. 
The problem is to derive the coefficient vector h that minimizes the MSE: 


SOLUTION: : 
The LMMSE estimator is expressed as 6 = m?,,M,'x. 
To derive the above result, we model the square form of the MSE with respect to h: 


MSE(h) = B| (@—h"x)? |. 


sé) =| (e-6)' |= | (0-h"x) | 
=h"Elxx"]h +E[6?]-h7E[ x6 |-Elox"]h. 


The vector h, which minimizes this square function, can be solved by factorization: 


ee aoe 
ah,’ ‘oh, 


= 2(hE[xx"]—- E[6x"]). 


07 = V,,MSE(h) = |mse) 


Therefore, the optimum vector h satisfies the equation: 


[x7 Th = E[x6]. 


Considering the matrix M, = E[xx’] is non-singular, this is equivalent to: 


st -1 
h=M,m,, 


and the optimal linear estimator is derived: 


AD — wl vngq-l 
6=m,,M, x. 


By substituting the above result in the expression of the MSE(h), the minimum MSE for 
linear estimators is defined as: 


MSE, nin = E[6?]—m1,M;'m,,. 


min 


It is noteworthy that, as the matrix My' is positively defined, the MSE cannot 
exceed the a priori second moment E[6?] of @. If the parameter has a zero mean, then 


[0°] = 


i[(O — 


\[6])*] = var(6), ie, the second moment is equal to the a priori variance, and the 
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LMMSE estimator performance is superior to that of the constant estimator 6 = E[6]=0. 
However, if E[6]#0 then E[6*]>E[(@—E[@])*] and the LMMSE estimator may perform 
worse than the constant estimator. Basically, this problem occurs because the LMMSE esti- 
mator is a biased estimator of 0, in the sense that its biased mean is E[0]— E[6] #0, except if 
i[8] = 0. The way to deal with this bias is by generalizing the category of linear estimators 
to that of the affine estimators. 


9.3 Optimal Affine Estimator of a Scalar Random Variable 0 


The affine estimator also depends linearly on the x-vector, but also includes a constant 
term to control bias: 


6=h'x+b=h'(x—E[x]) +c, 


where b and c=b+h’E[x] constitute a parameterization of the constant term and the 
remaining term, respectively. It is easier to use constant c in the sequel. Our objective is to 
derive the optimal coefficients j}h=[h,,---,h,]', cy. To implement an affine MSE minimiza- 
tion estimator, the knowledge of the expected values E[x], E[@] of the (invertible) covari- 
ance matrix is required: 


R, =cov(x)= al (x B[x])(x- ax])' |, 


and the cross-correlation vector: 


Tyg = COV(x,0) = | (x- a[x])(@ - A[9}) |. 


The problem is the derivation of the vector h and of the constant c that minimize the MSE: 


MSE(h,c) = 3| (0—h" (x- alxl)-<) | 


SOLUTION: 
The relation 6 = B[6]+1, .R;'(x—E[x]) represents the optimal affine estimator. 

To obtain the above desired result we again use the square form of the MSE with 
unknown h and c: 


MSE(6) = s|(o-6) |= i (O- c)—h" (x E[x)) | 
=h" il (x- a[x])(x- B[x]) |h+ B[(0 —c)?] 
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Note that the only dependence on constant c is via a term that is minimized by choosing 
c=E[6]. 
With respect to vector h, minimization occurs through factorization: 


0=V,MSE=h'R, -%., 
leading to the minimization equation for h: 
R,h=r,,. 
When the matrix is non-singular, this is equivalent to: 
h=R7'r, ¢, 


and the optimal affine estimator is expressed as: 


6 = B[O] +412 R;\(x—E[x). 


Unlike the linear estimator, the affine estimator is unbiased (for the mean value), in the 
sense that E[6]= E[6]. 
The minimum MSE obtained by this estimator is calculated as: 


MSE win = Var(0)— 1, 6R;'r, 9. 


x 


Thus, based on the bias handling, the optimal affine estimator has an MSE that never 
exceeds the value var(6), that is, the MSE of the constant estimator. 


9.3.1 Superposition Property of Linear/Affine Estimators 


Let y and g represent two RVs. The optimal linear (affine) estimator of the sum 0 = y+ 9, 
with conditional x, is defined as: 


A 


6=+9, (9.1) 


where y and @ are the optimal linear (affine) estimators of y and g, respectively, with a 
conditional x. 


9.4 Geometric Interpretation: Orthogonality Condition 
and Projection Theorem 
There is a deeper geometric interpretation of the structure for the affine or linear minimum 


mean square error estimators. In order to study this geometric interpretation, we re-define 
the problem of the affine estimation into a linear approach problem in the vector space. 


460 Digital and Statistical Signal Processing 


9.4.1 Reconsideration of the Minimum MSE Linear Estimation 


The key to embedding this problem into vector space is to locate the right space for the 
approximation problem. There are two kinds of space to consider: the space # containing 
quantities to be estimated (e.g., @) and the space S, called the solution space, in which the 
proper approach is structured (e.g., linear combinations of x;). The problem then involves 
deriving a linear vector combination in space S closest to the quantity we want to approx- 
imate in 1. For the efficiency of this approach, it is absolutely necessary for Sc ‘H to be 
true. Once we identify these spaces, what remains is the modeling of an inner product, 
which causes the appropriate norm that expresses an approximation error, such as the one 
of the MSE approach. 

In a similar basis as the minimum MSE problem, we try to approach the RV @ through 
a linear combination of measured RVs x,,---,x,, and we define H as the space that all scalar 
RVs with zero mean belong to and S as the linearly produced space span {x,, «+: ,x,,} of the 
measurements. For technical reasons, we will require all RVs in H to have a finite second 
moment (otherwise we can end up with vectors with infinite norms). 

The MSE between vectors 7,0 €H can be expressed as the squared norm: 


In-of =E[(n- v7], 


which results from the inner product: 


(n,v) =E[nv]. 


Since 6=h'x= ) h,x; is located in S space, the estimation of the linear minimum MSE 
i=1 
A a2 
for 0 is the vector 9 € S, which minimizes the squared norm lo - él : 


Linear Projection Theorem: 

The optimal linear estimator of 0, based on x,,---,x,, is the projection of in S = span { x, py Xp y 
Using the orthogonality condition in the projection theorem, the optimal linear estima- 

tor @ must ensure (Figure 9.1): 


(0-6,u,)=0,  i=1,--,0. (9.2) 


When {x;}j-1 are linearly independent, the dimension of S is equal to n and each basis of 
values must have n’ = n linearly independent elements. 
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S =span (Xz,...,Xp) 


FIGURE 9.1 
The ortogonality principle for the optimal linear estimator 9 of the RV 6, with conditional x,,---,x,,. 


Now we just need to adopt a specific basis to model the optimal linear estimator. Perhaps 
the most reasonable basis of values is the set of measurements themselves, u; = x,,1 = 1,--- 
(assuming they are linearly independent), so that: 


| (0-)x" |=0. 


Equivalently: 


| (@-h'x)x’ |=M",-h™M, =0. 


Since the linear independence of {x;} implies that the matrix M, =E[xx"] is invertible, it 
leads to the optimal solution developed earlier: h = M;'M,,». It is derived that: 


where, from the second equality and on, the fact that the optimal error is orthogonal to 
each vector of the space S was used. 
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9.4.2 Minimum Affine MSE Estimation 


The problem of minimum affine MSE estimation can also be transformed into a norm min- 
imization problem of a vector space. One way to do so is to subtract the average from the 
parameter and the average from the measurements, and proceed as in the linear estima- 
tion (adding the mean parameter back to the solution at the end). A more direct approach 
is to include the degenerate constant RV “1” in the measurement vector (we can always add 
a virtual sensor to the measuring system that measures a constant). 

To see how this could work, we first re-define the equation of the affine estimator as: 


6=h'x+b 
=[h’,b]| * |. 
| 


Next, the subspace of the solution is defined: 
S&4 span{x,,---x,,, 1}, 
with the help of which the following theorem of affine projection is presented. 
The Affine Projection Theorem: 


The optimal affine estimator of 0, based on RV x,,---x 
space span {X1,---X,,1}. 


is the projection of 6 to the produced 


ny 


9.4.3 Optimization of the Affine Estimator for the Linear Gaussian Model 


The additional assumption that x, 6 are jointly Gaussian distributed is introduced. It is true 
that the minimum MSE estimator is in fact affine: 


[0 | x] = B[6]+ 12 ,Rz'(x—E[x]). 


One way to show the validity of the above relation is to calculate the CME in such a way 
that then it can be expressed in the form of this relation. Alternatively, we can develop 
the following approach. Without loss of generality, let us consider the case of x and @ with 
zero expected values. Let the LMMSE estimator, which is the same as the affine estimator 
in this case. From the linear projection theorem it is known that the error of the optimal 
estimator is orthogonal to the measurement samples: 


| (0—6,)x |=0. 


However, since @— 6, constitutes a combination of linear Gaussian RVs, it implies that it 
is Gaussian distributed. Moreover, as the Gaussian RVs that are orthogonal to each other 
are independent RVs: s{ @ —6,)| x| = K[O—-6,]=0. 

Therefore, since 9, is a function of x, it is: 


0=E| (0-6,)|x |=E[6|x1-6,, 


Linear Estimation 463 


or 


[0 |x]=6,, 


that describes the desired result. 


9.5 Optimal Affine Vector Estimator 


When the parameter 0= [4,,---,6,]" is a vector, the aforementioned analysis for the scalar 
parametric values can easily be extended if the sum of individual MSEs is designated as 
the error criterion. We define the a priori vector of the mean value E[@] and the cross- 
correlation matrix: 


R,,9 = cov(x,@)=E|(x—E[x])(@—E[6])’ |. 


The error criterion of the MSE sum is defined as: 


P 
MSE(6) =)” MSE@,) 


i=1 
al 
M4 


6, -6; 


P 


* |= trace( [0-6 0-6)" }). 


i=1 


Let 6, be the affine estimator of the i-th element of 0: 
6,=h'x+b,,  i=1,---,p. 


The corresponding vector estimator is defined as: 


The problem of the minimum affine vector MSE estimation is reflected in the derivation 
of the appropriate H, b as the sum of MSE (Hb) is minimized. 
The solution to this problem is the optimal affine vector estimator: 


m 
Il 


[0] +R, ,R;' (x—E[x]). (9.3) 


x 
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The above relation is based on the fact that each pair of values h; and b; appears sepa- 
rately in each summand MSE(@). Therefore, minimizing the MSE is equivalent to minimiz- 
ing each MSE(6;): 


mja MSE(H, b)= ) min MSECh, b). 


Therefore, the solution for the minimum MSE is the sequence of multiple optimal scalar 
affine estimators for the individual 0;: 


6, z[9, | ty, Ry! (x — E[x]) 


6 1) |), .Rs@- EL) 


which is identical to Equation 9.3. 
The final minimum sum MSE can be defined as: 


MSE,,i, = trace(Ro—Ro ,R,'R,)- 


9.5.1 Examples of Linear Estimation 


The aforementioned results of the minimum MSE are applied in the following examples. 


Example 22: Linear Prediction of the Minimum MSE 
In the linear prediction, we consider that a part {Xp pt sXa} of the time sequence of sam- 


ples (eh. is being measured, aiming to the creation of a linear p-class prediction, 
which is in the form of: 


We also consider that {x,} is a zero-mean wide sense stationary random sequence, with 
an autocorrelation function: 


r(k) = Bl x; x; |. 


The problem lies (Figure 9.2) in deriving the prediction components a= [a,,---,4,]° 


that minimize the mean square prediction error: MSE(a) = El (x, = zy | 

To find the optimum solution, we set 6 = x, as the random scalar parameter, x, as the 
time interval that the measurement occurs and h = a, the vector of coefficients to be 
determined. 
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x(i) 


Previous data 


Filter FIR 


FIGURE 9.2 
The linear prediction mechanism as an FIR filter. 


Step 1: Define of the prediction equation in vector form. 
X, =a’x, 


where 


Step 2: Determine the orthogonality principle: 
E| (x, —a"x)x,.; |=0, i=1,---,p, 


or, in a vector form: 


B(x, - a’x)x,| 
0'= : = E(x; = alx)x" |. 


Bl (x, - al x) | 


Therefore, the optimal prediction coefficients a= a are depicted as: 


a=R'r, 
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where the correlation vector is introduced: 
r = [4%] = Elxx;], 


and the covariance matrix (Toeplitz): 


R= (1%, ;)i,j-1,p = E[xx’]. 


Finally, the prediction has a minimum MSE: 


="-a'r 
=%—-1R'r 
Example 23: Inversion Problem 
Consider the following measurement model: 
x=A0+4+n 


where 
e 0=[6,,-- ~0,)° are the unknown random measurements. 


var : : 
e n=[1,---,1,] is the the random noise of zero expected value and covariance 


matrix R,,. 
e Aisanm x p matrix with known elements. 


e 0,nare uncorrelated. 
‘T 
e x=[%,,---,X,] are the random measurements. 


The problem is to derive an affine estimator min MSE 6 of 0 (Figure 9.3). 


SOLUTION: 
From the previous results for the minimum MSE vector values, it is obtained that: 


6 = E[6]+ Ry, Ry (x—E[x)). 


We only have to determine the form of the optimal affine estimator as a function of 
A and R,; 


E[x] = E[A@+n]= AE[6], 
R, = Soy AG en) =AR,A‘™+R,, 


uncorrelated 


R,.9 = cov((A®@+ n),6)= ARg. 
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X1k 


Xmk 


Nmk 


FIGURE 9.3 
Block diagram of the inversion problem in an FIR filter assembly with p taps. 


Consequently, the final result can be expressed as: 
@= H[6]+RA7(AR,AT+R,) (x-AB[O), 
and the minimum sum MSE as: 
MSE rin = trace(Ry -R,A™(AR,A™+R, y ARy} 


Note that: 
1. When R,, dominates AR,A’, then MSE,,,;,, © trace(Ry). 
2. When AR,A’ dominates R,, and Matrix A is full rank, then MSE,,,,,, © 0. 


9.6 Non-Statistical Least Squares Technique (Linear Regression) 


In some cases, the model being measured is not sufficient for a clear calculation R and 
R, 4, which are the necessary statistics for the minimum MSE linear estimators described 
above. 

In these cases, we must turn to the training data for the estimation of these parameters. 
However, the following question arises: To what extent is it preferable to replace the empir- 
ical averages in the above formulas? The answer depends, of course, on the definition of 
optimization. The non-statistical least squares technique is a re-definition of this problem, 
for which the optimal solutions are in the same basis as the previous ones, but with empiri- 
cal estimates, by appropriately substituting R and R,.¢. 


468 Digital and Statistical Signal Processing 


Let the available pair of measurements (1 > p): 
Vis x; = [Xa Xp], i=1,---,n. 


where x;, may be equal to x;_,, but this is not necessary. 
Let the following relationship between input-output in the model (Figure 9.4): 


FE . 
Y,;=xXjat+0;, 1=1,---,n, 


where 
e y;is the response or output or conditional variable, 


¢ x;is the input or independent variable, 
¢ aisap xX 1 vector of unknown coefficients to be estimated: 


a=[4,,---,a,]’. 


Objective 
Derive the least squares linear estimator a of a that minimizes the sum of squared errors: 


n 


sse(a)= }°(y;-x7a) 


i=1 


This model is expressed inn X 1 vector formulation as: 


yy xy O71 
e tel > lag) 2h 
T 
Yn Xn On 
y=Xatv, 


where X is a non-random n x p input matrix. 


x(k) 


y(k) 


FIGURE 9.4 
Block diagram of the linear regression. 
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The estimation criterion is expressed as: 
SSE(a) = (y—Xa)'(y—Xa). 


SOLUTION: 
Step 1: Determine the vector space containing vector y: H = R". 
Inner Product: (y,z)=y"z- 


Step 2: Determine the subspace that defines the solution of the problem, which contains 
Xa. 


S =span{columns of X}, 


which contains vectors of the form: 


p 
Xa= a [xyes Xue | « 


i=1 


Step 3: Apply the projection theorem. 
Orthogonality principle: The optimal linear estimator a satisfies the following relation: 


(y-Xa,u;)= 0, = 1,1, 
where u; are the columns of matrix X, or equivalently: 
0” =(y-xa)' x 
=y'X-a'X'X, 
or, if matrix X is of full p-rank, then XX is invertible, and: 
=1 
a=[x'x] x’y 


=[nx"x] [nx ]y 
=Re 


x x,y" 


It is true that: 
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The form of the projection operator is then determined to predict the output response as: 


A 


y=Xa, 
where, as before, it can be expressed as an orthogonal projection of y to S: 
y=X 


=Xa 
= X[X'X]'X’ y. 


orthogonal projection 


Properties of the orthogonal projection operation: 
Tl, = X[X’X]'X". 
First Property: The operator |], projects vectors to the column space of X. 


We define the decomposition of the vector y in the coefficients y,. in the column space of X 
and yx, which is orthogonal to the column space of X(Figure 9.5): 


Y=YxtYx- 
For a vector a=[a,,---,a,]', it is true that: 


Yx=Xa, X'yx=0. 


Projection of 
column X 


FIGURE 9.5 
The decomposition of vector y in the column space. 
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Thus: 


Thy =x (yx a y') 
=X[X'X]'X'Xa+X[X'X]'X'yx 


=I =0 


=Xa 
=Yx, 


so that the operator [], derives the column space for the coefficients of the y-vector. Thus, 
y, = ILy can be determined such that the following result is obtained: 


y=Iky+(I-M)y. 
yx 


Second Property: From the previous property it is true that the operator I — [], is projected 
into a space orthogonal to span{columns of X}. 


Third Property: The operator [I], is symmetric and idempotent: T}I1, =I. 
Fourth Property: (I — Ty) Tx = 0. 


Example 24: Least Squares Optimality of Sample Mean 


T 
Let the measurement: x = [ ey aisle =| ; 
Objective: Derive the appropriate constant c, which minimizes the sum of the squares: 


Y Gi-0)? = ct)" (xc), 


where 1 = [1,---, 1]. 


Step 1: Define the solution space. 
Space S is the diagonal line: ty :y=al, ae R}, as illustrated in Figure 9.6. 


Step 2: Apply the orthogonality principle. 


7 n 
1 
(x-cl)'1=0erc= 75 =n" y X33 
k=1 


Example 25: Let the Measurement Sequence {z;} 


The sequence of training data consists of n + p samples of z;: 


{z}et, i=1,---,n. 
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FIGURE 9.6 
The diagonal line is the subspace produced for the scalar least squares case. 


Using an auto-regressive model for the training data process, it should be true that: 
P 
Zp = Saiz: +u,, k=p+1,--n, 


i=1 


so, as the SSE gets minimized (Figure 9.7): 


2 
SSE(11) = Sn - Seem 
i=1 


i=1 


SOLUTION: 


Step 1: Determine response variables y, = z, and input vectors 2, =[Zj_1,°°", Zp] : 


Zn+p Zn+p Onsp 
=}: fat) oo: |, 
T 
You Zp+1 Ons 
y=Xatv, 


Step 2: Apply the orthogonality principle. The prediction of p-th order LLS is of the 
form: 


ZR = » AjZ Kis 


P 
i=1 
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x(i) 


Prediction 


Previous data 
X(k) 
Filter FIR 


FIGURE 9.7 
Implementation of the LLS prediction from a sequence of training data. 


where vector a=[@,,-:-,4,]’ is derived by the equation: 
a=[X"X] 'X’y=R“4, 


We also introduce the following quantities: 


ae Eee 
R=(Fi-j))i pap, 


ee _ 
Gan Sy Fis j=0,---)p. 


Lae 
9.7 Linear Estimation of Weighted LLS 


As before, we consider a linear model that expresses the input-output relation: 
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The linear minimum weighted least squares-LMWLS estimator a of a minimizes the 
following relation: 


SSE(a) = (y — Xa)" W(y— Xa), 


where W is a symmetric positive defined matrix, n x n. 


Solution to LMWLS problem: 
Step 1: Define the vector space that contains the vector y: H=R”. 
Inner product: (y,z)=y"Wz. 


Step 2: Define the subspace S that constitutes the solution of the problem: 


Xa =span{columns of X}. 
Step 3: Apply the projection theorem. 


Orthogonality principle: The optimal linear estimator a satisfies the following relation: 


0" =(y-xa) WX 
=y'WX-a'X'WX, 


or, if the matrix X is full p-rank, then X’WX is invertible and: 
a=[x’wx] X"Wy. 


Alternative interpretation: The vector y of the least square predictions y;=x/;a of the real 
output y is: 


which represents a non-orthogonal (oblique) projection of y in space H (Figure 9.8): 


y=Xx[x’wx] x"Wy. 


oblique projection ITx w 
Then, the weighted sum of square error (WSSE) is derived: 


WSSE win =" | I-X[X™WX]?X"W || I-X[X™W x}'x'w] y 


= y [1- Iw a [1- Tey 192 
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Projection 
of column 
X 


FIGURE 9.8 
Oblique projection of WLS estimator. 


The LMWLS prediction technique can be depicted as a linear least squares (not weighted) 
prediction on the status of the pre- and post-processing data (Figure 9.9). 

As the matrix W is symmetric positive definite, it is subjected to a quadratic root factor- 
ization that is of the form: 


W= Ww!2wl/2 } 
and 


y= WwW??? wi?x [x"wi?w'?x T xtw? [wy | 


orthogonal projection IT 


w" *y. 


wi/2x 


=W TI 


w!?x 


Example 26: Adaptive Linear Prediction. 


Let us model the auto-regressive p-order model: 


p 
Zp = Vaz i+ 0, k=1,2,... 


i=1 


so that, at time n, the weighted least squares criterion is minimized: 


2 
n P 
WSSE(n) = ye [.., ~ Setup y 


k=1 i=1 
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FIGURE 9.9 


Implementation of the LMWLS estimator using pre-processing and post-processing units for orthogonal 


projection. 


and the coefficient p € [0,1], as illustrated in Figure 9.10. 


SOLUTION: 


As before, we define the response variables y, = z, and the input vectors, 


2 = [Zpa7* ifesl We also define the weighted matrix: 


p 0 0 
W=) 0 =. 0 
0) 0 p”! 


In this way, the prediction coefficients for LMWLS are derived as: 


a=[X’WX]'X’Wy 
-1lA 
r, 


A» 


where the smoothed correlation coefficients of the samples are given: 


FIGURE 9.10 
The coefficient p € [0,1]. 
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F 
R=(ri-j) 


i,j=1,p’ 


n 
5 A n-i a 
7. PP’ ZinpZiap-jrp J =O, p- 
i=l 


The minimum weighted sum of the square error (WSSE) is expressed as: 


ae 
9.8 Optimization of LMWLS in Gaussian Models 


We know that the linear MMSE (LMMSE) estimator is the best among all others (linear 
and non-linear) for Gaussian models with respect to the measurement samples and the 
parameters. In this section, a similar case of the linear WSSE estimator is studied. 

Let the following Gaussian model: 


y=Xatv, 


where we consider that 
e v~N,(0,R). 
e The correlation matrix R is known. 
e The matrix X expresses the known and non-random measurements samples. 


Based on the aforementioned model, for each conditional matrix X or vector a, the PDF 
of vector y is a Gaussian multivariable given by: 


fly;a)= A op( ly Xa)"R(y-Xa)} 
(27)" RI 2 


This implies that the maximum likelihood (ML) estimator of ais the same as the LMWLS. 
This is more clearly stated by the following relation: 


yy, = arg max, In f(y;a) 
=argmin,(y—Xa)'R'(y—Xa). 


Consequently, 


y=Xa,, =X[X'’R'X]'X’R“y=Nk wy. 
y ML y xwy 
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With reference to this model, we can evaluate its performance by examining whether it 
satisfies the equality condition of CRB: 


(V,In fy" =(y—Xa)"R2X =| y"R*X[X'R?X] —a" |X™R"X 


a 
al 


We conclude that when the matrix X is non-random and the (known) noise covariance 
matrix R is equal to the weighted LS matrix W~1, then the following conditions are true: 


e The LMWLS estimator a is unbiased. 
e The LMWLS estimator is efficient and, consequently, UMVU. 


e Based on the property 5 of the CRB (Chapter 8), since K, is not a function of a, the 
covariance of the estimator is: 


-1 al 
cov, (a) = Kj! = [x’R'x] aR 
nN 


10 


Fundamentals of Signal Detection 


In this chapter, the problem of signal detection is analyzed. This is equivalent to the prob- 
lem of the estimation studied in the previous chapters if we consider that there is a limited 
range of possible values of the unknown parameter 8. For this reason, the theoretical con- 
cepts of detection are generally simpler than the concepts that construct the problem of 
estimation. As discussed below, these theoretical concepts are related to each other in the 
case of unknown nuisance parameters. 

In this chapter, the following concepts are discussed: 


Optimal detection theory 

Detection using the Bayes approach 
Frequentist Approach for Detection 
Receiver Operating Characteristic (ROC) 
Multiple hypotheses testing 


Example 27: Typical Radar Application 


Let a continuous time x(t) be over the time interval [0, T]. We want to decide if x(f) con- 
tains only noise: 


xH=w(t), O<tsT, 


or if there is a signal along with the additive noise: 


x(t)=Os(t-t)+ w(t), OStsT. 


The following apply: 


¢ s(f) is a known signal, which may be present or not. 

¢ w(t) indicates white Gaussian noise with zero expected value and known level 
of power spectral density N,/2. 

e tisa known time delay,0<t<«T 


ce fa 
° fise = a) dt =[ swf dt is the signal energy. 
0 0 


e 6€ [0,1] is an unknown nuisance parameter. 


The objective of the detection process is (a) to decide on the presence of the signal or not 
and (b) to make the decision with the minimum (average) number of errors. 
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We will use the following symbols for the detection case: “Signal Presence -H,” and 
“Signal Absence -H)”: 


Hy: x(t) = w(t) Hy:0=0 


H,: x(t) = s(t-T)+ w(t) H,:0=1. 


Two typical methods for detecting signals are illustrated in Figure 10.1 and are the 
following: 


e Energy threshold detection: 


T Ay 
= flerae > 
y= J at > 


Ho 


x(t)=s(t-t)+w/(t) 


Hy 


x(t)=s(t-t)+w/(t) 


(? Y 
t=T 
FIGURE 10.1 


(a) Energy threshold detector, (b) threshold-filter detector. 
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e Threshold-Filter Detection: 


Ay 


y= frcr ~#x(t)dt > 7. 
< 


Ho 


Error analysis: 
According to Figure 10.2, there are two categories of the decision error: 


e False alarm: y > 1 when there is no signal presence. 
e Miss detection: y < n when there is a signal presence. 


We can easily calculate the conditional probabilities of these error cases, respec- 
tively, as: 


P, = P(signal detection| no actual signal) = (y | no actual signal) dy 
gna gna y gna 


yon 


Py = P(non signal detection | signal) = } fly | signal) dy 


ysn 


y(t) 


Miss detection False Miss detection False 


FIGURE 10.2 
Repeated radar tests in received sequences y;. One sequence contains a signal (H,), while the second one con- 
tains its absence (H,). 
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The following two fundamental questions arise: 


Q1: Is there an optimal way of matching Py and P,? 


Q2: Can the filter h() in the threshold-filter detector be optimized to provide the opti- 
mal tradeoff? 


The answer to Q1 will be given later. At this point, the filter of the threshold-filter detec- 
tor can be designed in an appropriate way so as to optimize a given design criterion. As 
shown in Figure 10.3, there is a relatively large overlap between the two PDFs. This fact 
should be taken into account in the design of the optimum filter in order to minimize this 
overlapping. One measure of the degree of this overlap is the deflection, which is defined as: 


| D[ y [signal presence |- | y [signal absence | 


ae 
var(y | signal absence) 


Large values of d? are translated into well separated PDFs f(y|H,) and f(y|H,), with little 
overlapping. Thus, by increasing d?, the overlap is reduced. 


F(y | Ho) F(y | Hi) 


Pm 


FIGURE 10.3 

Likelihoods of miss detection (P,,) and false alarm (P,) for the radar example. We notice that the reduction in 
the error of one of the two PDFs (by changing the decision threshold 1) causes an increase in the error of the 
other and vice versa 
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Deflection can be easily calculated for the radar example. We notice that the presence of 
a signal produces some shift in the expected value, but not in the variance: 


ily | signal absence] = 0, 


vi 
E[y | signal presence] = [rer —t)s(t—7)dt, 
0 


Tr 
var[y | signal absence] = 2 } On dt. 
0 
Thus, be applying the Cauchy-Schwarz inequality, it is derived that: 


2: 


T 


Jucr—pste- aya 
0 


# = : <= fice a) dt, 
0 0 
[mcr —1) dt eo 
0 JiscoP arlsl? 


where ‘’ is true if and only if h(T—t) = as(t- 7) for a constant a. 
Consequently, the optimal deflection filter is the matched filter: 


h(t)=s(T +T-2). 


If s(z) is a short duration pulse: 


aE 
e fie - a), dt does not depend on t. 
0 


e The optimal detector is expressed as: 


y= | st—7)x(f)dt 


oo 


= Jse- 1) x(t) dt 


—co 


= s(-t) * x(f)|,_.- 


In Figures 10.4 and 10.5, the matched filter detector is illustrated in different signal cases. 
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Hy 
x(t)=s(t-t)+w/(t) 


S(T+t-t) 
t=T 


Ho 


FIGURE 10.4 
Application of the matched filter to the optimal receiver when receiving a delayed signal and noise. 


Hi 


x(t)=s(t-t)+w/(t) 


s(t-t) 


FIGURE 10.5 
Application of the correlated receiver to receive delayed signal and noise. 


10.1 The General Detection Problem 


First of all, the following terms are defined: 


e xe isa RV, the sample to be measured. 
e @€@is an unknown parameter. 
© f(x; 0) is the PDF of the RV x, which is considered known. 


We have two discrete hypotheses regarding parameter 0: 
6€0O,) or OEO,. 
©,,©, form a partition of © in two non-overlapping regions (Figure 10.6): 


Q,UO,=8, 0,0, ={}. 
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FIGURE 10.6 
The detector decides on the range of values of © which contains the possible values for the parameter 0. 


It is true that: 


Hy :8€0, Hy:x~ f(x;0), 0@€0) 


H,:0€0, Hy:x~ f(x;0@), @€0). 


where: 


e H)is the null hypothesis and the existence of only noise. 
¢ H, is the alternative hypothesis with the existence of both noise and signal. 


10.1.1 Simple and Composite Hypotheses 


When the 6 parameter can only get two values and ©), ©, are monotone sets, the hypoth- 
eses are called simple: 


©={0,,0,}, Q,={A}, ©, ={6)}. 


In this case, the PDF f(x; @) is fully known with either H, or H, being conditional. 
If the hypotheses are not simple, at least one of ©), ©, is not a monotone set and, in this 
case, it is called composite. 


486 Digital and Statistical Signal Processing 


10.1.2 The Decision Function 


The purpose of detection is to efficiently design the decision rule (test function): 


1, decide H, 
(x) = 
0, decide Hy 


The test function g(x) assigns the field ¥ to the decision area {0,1} for the choice of H 
and H,, respectively. More specifically, g(x) divides ¥ in decision areas (Figure 10.7): 


X= {x :Q(x)= o}, X= {x :Q(x)= 1}. 
Errors of miss detection and false alarm: 


The probabilities of false alarm and miss detection associated with the test function g are 
expressed as: 


P= Bylol= | ox) flxie)dx, 0€0,, 


Py =Bell-9]= | (1-912) f(xie)dx, 0€,. 


Equivalently, it is true that: 
ae | feloax, 9€®,, 
xX 


a | ferloyax, 0€®,. 


XY 


O(x) 


FIGURE 10.7 
The test function splits the measurement space in two decision areas. 
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The probability of a right decision for H, is called detection probability: 


P,(0)=1-Py(@)=E,[g], 9€9,, 


and is illustrated in Figure 10.8. 
Figure 10.9 illustrates the above metrics. 
Next, the cases of random or non-random 6 are examined separately. 


fly | Ho) fy| Hi) 


FIGURE 10.8 
The probability of detection Pp = 1- Py 


FIGURE 10.9 
Graphic representation of decision areas 4,,4, with respect to decisions H,,H, for an observation of RV x. The 


probability of a false alarm, P;, is given by the integral of f(x;0o) in X,, the probability of miss detection, Py, is 
given by the integral of f(x;0,) in 4, and the probability of detection, P,, is given by the integral of f(x;0,) in ,. 
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Le 
10.2 Bayes Approach to the Detection Problem 
In order to implement the Bayes approach, the following must be considered: 


1. Assign an a priori PDF f(@) for 0. 


2. Assign cost or risk to miss decisions: c, = cost when H; is decided, but H; is really 
true. 


3. Find and implement a decision rule that exhibits the minimum average risk. 


10.2.1 Assign a Priori Probabilities 


The a priori probabilities for H),H, are defined as: 


P(H,)=P@€®.)= | f(6)d6, 


®o 


P(H,) = P(6€®,)= i f(0)d0. 


with P(H)) + P(A,) = 1. 
The calculation of conditional PDFs, of Hj and H,, is derived by integration at 0: 


[ feloferae 


f(x|Ho)= : P(H,) , 
[ flo feyae 
9; 

fo) = "pe 


10.2.2 Minimization of the Average Risk 


We define the cost or risk matrix as follows: 


cs C11 Ci9 
Cor Coo 


We consider ¢;; < ¢;, i.e., the cost of a right decision is less than the cost of a miss decision. 
The actual cost for a given RV x, denoted as C, is a function of (a) the output g(x) of the test 
and (b) the real case H, or H,. Consequently, the cost C € {c11,C19,CoyCoo} is a RV and thus, 
we are looking for decision rules that minimize its average value, also called the “average 


risk” associated with the decision function. 
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We accept the following Bayes criterion: 
Choice of the appropriate gy (equivalently X),%,), that minimizes the average risk, which 
is equal to the statistical expected value E[C] of the cost, C: 


S[C] = c,, P(A, | H,)P(A,) + copP(Hy | Ho) P(A) 
+ Cy9P(H, | Hy)P(Ho) + Co:P(Ao | H,) P(A). (10.1) 


Definition of Bayes’ false alarm and miss detection: 


Pe = | fx |H)dx = PCH, Ho), 


xX 


(10.2) 
Py siie [ folHyax = P(H,|H,). 


x 


The above expressions differ from the probabilities P-(@) and P,,(0), as mentioned earlier, 
as they represent error probabilities involving integration of 0 at ©) and ©,. Therefore, 
Relation 10.1 can be expressed as: 


SLC] = c,,P(A,) + cogpP( Ho) 
+(Cy9 — Coo) PCH) Pr + (Cor — C11) P(A) Pu- 


NOTE: E[C] is linear with respect to Pyy,P,,P(H,),P(Hy) for any decision rule ¢. 


10.2.3 The Optimal Bayes Test Minimizes E[C] 


According to Relation 10.2, E[C] can be expressed as a function of the decision area 4: 


ELC] = cy, P(A) + CogpP(H) 
+ [[ten ~ Cop) P(Ho) f (2H) = (Cor — es )P(H) f (x |) Jax. 


XY 


The solution is now obvious: if we had to choose to assign a point x to x to Y, or %, then 
we would only choose 4, if it reduced the average risk (i.e., it would make the integral 
negative). Thus, we assign x to 1, if 


(ne Cy) P(Ho) f (x|H) < (Co - C1) P(H,) f (x|H), 


and x to X, otherwise. 
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When Cy) > Cog and Co, > Cy, the optimal test is expressed by the Bayes likelihood ratio 
test (BLRT): 


Vv 


Ag(a) 2 Lee 
f(x|Hp) 


where vis the optimal Bayes threshold: 


— (C10 = Coo) P(A) ; 
(Cor — C1) P(A) 


We notice that the costs and the a priori probability p = P(H)) = 1— P(H,) affect the BLRT 
only via the threshold 7, since the BLRT does not depend on p. 


10.2.4 Minimum Probability of the Error Test 


Let the special case, where Cg) = Cy, = 0 and Co; = Cy) = 1. This changes the average risk of the 
probability error criterion: 


S[C] = PyP(H,)+ PpP(Ho) = P., 


which is minimized by the likelihood ratio test: 


fel) > POH) 
F(x|Ho) < P(A) 


Using Bayes’ rule, we can easily see that it is equivalent to the maximum a posteriori 
(MAP) test: 


Ay 
P(H, |x) > 
P(H\x) < 


10.2.5 Evaluation of the Performance of Bayes Likelihood Ratio Test 


For the sake of simplicity of symbolism, let C £ E[C]. Consider C* to be the minimum 
average risk derived by the BLRT: 


C* = cy, P(H,) + CopP(H) 
+ (Cyq — Coo) P(g) Pe (1) + (Cor — C11) PCH) Pa (0). 
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0d 1 P=P(Ho) 


FIGURE 10.10 
The minimum (average) risk curve, corresponding to the optimal BLRT, describes a realizable lower threshold 
at the average risk of any test 


where 
P¥(n)=P(Ag>1/Ho), Pi(n) = P(Ag Sn\H,). 


Interpreting C* =C*(p) as a function of p, the minimum average risk is represented as the 
performance curve illustrated in Figure 10.10, which is a function of p = P(H,). Note that the 
average risk of any test is linear at p. The minimum risk curve describes a lower threshold 
at the average risk that is encountered to any test and any value of p. 


10.2.6 The Minimax Bayes Detector 


In many cases, the real value of p is not known. Therefore, the BLRT optimal threshold 
cannot be applied. Since all tests present a linear average risk, this average risk may take 
very large (unacceptable) values as p approaches either 0 or 1. This is represented by the 
straight line in Figure 10.11. It is true that: 


Cia = max C(p). 


pe[0,1] 


It is evident from Figure 10.11 that the minimax test is the optimal Bayes test (i.e., a test with 
its average risk line tangent to the minimum risk curve), which is implemented using the 
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Max Cy (p) 
p 


Cy+(p) 


Co*(P)=Crinmax(p) 


C*(p) 


p* 1 p=p(Ho) 


FIGURE 10.11 
The curve of each test g is a straight line. The optimal minimax test ¢* is depicted by a horizontal line that is 
tangential to the minimum risk, named C*(p), at its maximum value. 


threshold 7* that causes C to be a horizontal line (the slope of C must be zero). Therefore, 
we have the following optimization minimax condition: 


=0 


Cy (1- P#(n))+ CoPu(n), 


where P*(n) and Py;(n) are the Bayes probabilities for the false alarm and the miss detec- 
tion respectively, of the BLRT with threshold 7. 

In the specific case that C= P, : Cg) = Cy, = 0, Cy = Co; = 1, the minimax condition for the 
MAP test is derived, which is expressed as: 


c=[Pim— Pic |p+ Pic. 


=0 
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This implies that the threshold 7 should be chosen in a way such that the “equalization” 
condition is satisfied: 


P#(n) = P(Ag > Hp) = P(Ag $ 9H) = Pa(n). 


By substituting the minimax value of 7 with n*, and by selecting the appropriate thresh- 
old by the choice of a value of p, the minimax threshold is then related to the choice of p* 
through the condition n* = p*/(1 — p*). 


10.2.7 Typical Example 
Example 28: A Radar Application 


Let a known output of the matched filter, y. The optimal Bayes estimator is to be derived. 
Let P(H) = P(H;) = 1/2. It is true that: 


y= fooxw dt, 
0 


which constitutes the realization of a Gaussian RV with an expected value and variance: 


vu 
Ely|H)1=0, var(y|H,) = A {sco dt = 02, 


0 


T T 
Bly|H,]= flew? dt=p,, var(y|H,)= Xs {sof dt =0?. 
0 0 


The likelihood ratio test is given by: 


ay“ 
1 o 208 4 
2104 
A3(y)= To 
208” 
2no; 


1 My 

Yi, /09-SHt/08 > 
=e 2 n=1. 

< 

Ho 
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Also, the statistic of the likelihood ratio test A,(y) is a monotonic function of y, since 
Ht, > 0. An equivalent test is via the threshold detector-filter: 


For the example discussed here, the Bayes test performance is described as: 


P; = P(y > y|Ho) 


= P(y/0y > ¥/0o|Ho) 
NO) 


=1-N(7,00) = QAy/05), 


and 


Py = Ply <y|Hy) 


= (y= ma)/ oe >(¥=H1)/ o> \#) 
N (0,1) 


=N((7-14)/o): 
NOTE: Since the standard Gaussian PDF is symmetric M(—u)=1-—N(u) and y = ,/2, thus: 


Py = Pr =1-N(t,/(20,)). 


We conclude that the Bayes threshold is minimax. Therefore, the error probability is 
reduced as follows: 


P= Py? +P? 
= Pp, 


Figures 10.12 through 10.14 illustrate the above results. 
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CDF 


PDF 


N(-u)=1-N(u) 


FIGURE 10.12 
The CDF N(u) of the Gaussian distribution. 


U1 


P=1/2xy, 


FIGURE 10.13 1 
Hypotheses of equal likelihoods, which have a minimax threshold y= ahh for the shift detection problem of 
the expected value of a Gaussian RV. 
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Pe 


1-Fy(0)=0.5 


1-Fy(1)=0.15 


Oo 


FIGURE 10.14 
The probability error curve of the Bayes likelihood ratio test as a function of , = |[s||?. 


10.3 Multiple Hypotheses Tests 


The objective is to measure the RV x, which has a conditional PDF f(x|0), with 0 € ©. Also, in 
this section, we consider that the total parametric space is divided into subspaces ®,,---,Oy). 
For the test of M hypotheses for 0: 


the decision function is defined as: 
T 
(x) =[9,(2),-- Ou] , 


where 


a(x)e(0,U, > 9(2)=1. 


i=1 


NOTE: The decision function determines the partitioning of the measurement area ¥ in 
M decision areas (Figures 10.15 and 10.16): 


X={x:9,=1, i=1,--,M. 
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The Bayes approach is based on three factors: 


1. Assign of ana priori PDF f(@) for 0. 
2. Assign cost to miss decisions: c;, = cost when H; is decided, but H; is true. 


3. Derive and implement a decision rule that presents the minimum average risk. 


10.3.1 A priori Probabilities 
The a priori probabilities of H; (i=1,---,M) are defined as: 


P(H,) =PO<e,)= | f(0)d6, 


i 


M 
with }" P(H,)=1. 
i=1 


The corresponding conditional PDF are derived: 


[reo fe@ae 
felE)="—say 
thus, the above composite hypotheses are transformed in the following simple hypotheses: 

A, :x~ f(x|Hy) 
Hy: Xx se 


where H; has ana priori probability P(H)). 


10.3.2 Minimization of the Average Risk 


The dimensions of the cost or risk matrix is M x M, and the matrix is given by: 


Design criterion: 
Choose the appropriate @ (alternatively, {4;}*,) so as to minimize the average risk E[C]=C: 


M 
C=) c)P(H, 


i,j=1 


H,)P(H,). 
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Next, the specific case is studied, where: 


© cz=0, 
© = List] 
1] 


In this case, it is true that C= P3 


H,)P(H,) 


i, j:i4j 


=1- - P(H,|H,)P(H;) 


i, p:i=j 


H,)P(H;) 


1, f=] 


: 1 F fronmie 


i=1 x; 


In order for C to get the lowest possible value, it should be true that: 


xeX, & f(x|H,)P(H;)2 f(x|H,)P(A;), 1#], 
or, with respect to the decision function: 


1, f(x|H;)P(A;) 2 f(x |H;)P(A;) 


p(x) = ‘ 
0, f(x|H;)P(H;) < f(x|H))P(A;) 


Introducing the following symbolism for simplicity: 
H, = H,(x)=argmax,, { f(x|H,)P(H))}, 
we can see that it is equivalent to the MAP rule: 
H, =argmax j;, {P(H; zyh, 


NOTES: 


e¢ The MAP decision rule minimizes the average error probability P.,. 


e The average error probability P, is equal to: 


P,=1- )°B[ 9((x)|H, JPA). 
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e The MAP decision rule for the RV x depends only on the likelihood ratio (which 
is statistically sufficient). 


¢ For equiprobable H;, P(H;) = 1/M and MAP are of the form: 


H, =argmax,, { f(x|H,)}, 


which can be also expressed as: “the estimation H, =H, if f(x|H,) >f(x|H))”. This expresses 
the estimation of maximum likelihood of the true hypothesis H;. 


Example 29: Classifier of Gaussian Average Values 


Let x = [x,-+,x,]? be an IUD RV N(1,0°) with a known variance o?. 
Objective: Classification of 1 to three possible values: 


Ay: =, 
Ay: W= bb, 
A: = Hs, 


considering equiprobable hypotheses. 
It is known that the MAP classifier depends on x, only via the statistic sufficiency for ju: 


which is a Gaussian RV with expected value, y, and variance, o7/n. 
Thus, the MAP test is of the following form: 
Decision of the hypothesis H;, if and only if: 


FO | Ai) = fe | Ay), 


where 


f(x|A,)= 


: | 1 _& | 
One Te Pp ie Th ky |, 


or, alternatively, after some cancellations and taking the log of the above relation: 
= 1 = 1 
Thy — 5 Me 2 XM My 


We conclude that these are linear decision areas. 
Numerical example: Let i, = — 1,y = +13 = 2. 
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FIGURE 10.17 
For three hypotheses of the Gaussian average, the corresponding decision areas are determined by the intersec- 


tions of three lines y = X My - —e ,k=1,2,3, which are depicted as a function of x. 
2 


We design three lines as a function of X in order to locate the decision areas: 


These areas are separated by specific levels in Y =]R" (Figure 10.17). 


10.3.3 Disadvantages of Bayes Approach 


The disadvantages of this approach are: 


e A priori knowledge of 6,H,,H,,... is required. 

e It ensures only the optimal average performance with respect to the a priori cho- 
sen parameter. 

e It does not guarantee protection of the metric of the false alarm and the miss 
detection. 
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10.4 Frequentist Approach for Detection 


The Frequentist approach is not based on a priori knowledge of H),H,. Consequently, the 
average probability of error or risk cannot be defined and minimized thereafter. Thus, an 
alternative criterion is adopted: limitation of the false alarm (up to a given value) and then 
minimization of the probability of miss detection. It is derived that, in order to find the 
optimal test satisfying this condition, we need to extend the aforementioned definition of 
the test function g, so as to allow random decisions: 


1, estimate H, 
@(x)= 4 q, probability of H, = q. 
0, estimate Hy 


We note that the above relation is interpreted as: 
¢(x) = P(H,| given the observation of x). 


The probabilities of false alarm and miss detection are functions of 0: 


P-(0), @€O), 
P5(0), @EO,. 


Yel Q] = Joos (x;0)dx = | 


Definition: A test ¢ is at the level a €[0,1] regarding the false alarm when: 


< 
mex P, (0) < a. 


Definition: The power function of a test is given by: 
B(0)=P,(0)=1-P,,(0), 00. 

10.4.1 Case of Simple Hypotheses: 6 € {4 ,0,} 

Let: 


Hy: x ~ f(x;0), 
Hy:x~ f(x;9,). 


We follow the Neyman-Pearson strategy, which detects the most powerful (MP) test g* of level a: 


1, L9*] = Ey, lel, 


for all trials that satisfy the condition E,,[@] <a. 
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Lemma Neyman-Pearson: The MP test of level a €[0,1] is a random likelihood ratio test 
of the form: 


= 


, f(x;0,) > f (x74), 
p* (x)=) 94, f(x:6,)=1 f (x7), (10.3) 
,  f(x;8,) <1 f (x; 6p). 


j=) 


where the 7 and q are appropriately chosen to satisfy: 


Lg, Lp*] = 4. 


Proof: 

First way: 
We use the Karush-Kuhn-Tucker method to maximize under constraints. 

The MP test mentioned above maximizes the term E,, [ e(x) | under the constraint 
X65 L(x) $a. This estimation problem under a constraint is equivalent to maximizing 
the following objective function without constraints: 


L(p) = Ey, Le(x) ]+A(a— Ea, [9(2)]), 


where A > 0 is the Lagrange multiplier, which is chosen so that the resulting solution g* 

satisfies the equality relation to the aforementioned constraint (i.e., Eg, [ p(x) | =a). 
Consequently, the resulting power can be expressed through the transformation of the 

expected value of the likelihood ratio, also known as “Girsanov representation”: 


” _F f(x;4,) 
de, L(x) |= | cf 2a | 


and thus: 


£(9) 8 By oon( fee - i)|+ae 
0 


According to the above relations, for a random realization of x, we only set g(x) = 1 
if the likelihood ratio exceeds the value i. If the likelihood ratio is less than 4, we set 
g(x) = 0. What remains is the case where the likelihood ratio is equal to i. In this case, 
we make a random decision (set g(x) = 4,0 < q < 1), so that the desired level of false alarm 
is satisfied. 


Second way: 
For any value of 9, g* satisfies: 


Hi, Lp*] 2 Ee, [p], when Ey,[9*]=4, E, [p] <a. 
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Step 1: 
MW hon Been 


By listing all the possible cases “>”, “<” an among the terms of Equation 10.3, we have: 


p* (x)[ f(x:61)—n f (x3) |= P(x) F(x.) — 1 f (x79) |. (10.4) 


Step 2: 
We integrate Relation 10.4 to all possible values of x: 


Jor co fie.) F006) Jax | obo F(:0,)—n Fai8,) Jax 


= [or @fxiedx—n | 9* 9 fles6)dx 
° Ey, [o"l : Eg, l9'l 
> | 9(2) flxi0,)dx—n | o(x) Fi6)dx. 


Eg, [¢] Egle] 


Hence, 


Jo,[9*]- ialol=n/ Be, [p*]- Bale) =0 


which proves the result of Neyman-Pearson Lemma. 
General remarks concerning the MP tests: 


a. The definition of the likelihood ratio test: 


f(x;0,) 


A f(05) < 


1. 


b. The probability of false alarm of the MP test is (A = A(x)): 


P; = Bo, 9*(x)]=P,,(A>m)+9P,(A=n). ao 
1-F, (nlHo) : 
“Randomization” should only occur when it is impossible to find the appro- 
priate value n, so that P, (A >n)=a. This is achieved only in the case where the 
CDF F,(t|H)) has jump discontinuities, ie., when there are points t > 0, where 
P(A =t)>0 and A = A(X) is not a continuous RV. Otherwise, g = 0 and therefore, 
such randomization is not necessary. 
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1—Fa(t| Ho = Pa, (A>d 


FIGURE 10.18 
Randomization is necessary to achieve level a, when 1- a is not within the CDF domain of A. 


When an appropriate value of n cannot be derived, so as to make P, (A >n)=a 
true, the implementation process is described as follows (Figure 10.18): 
¢ We derive the smallest value of ¢ for which the probability P,,(A>1) is less 
than a when a jump discontinuity is detected in the CDF. We set this value as 
a- and the threshold n equal to this value of t. 
e We define a* = P,,(A=)+a, where a and 7 are already defined above. From 
Relation 10.5 it is true that for any q value, the test has the following false alarm 
probability: 


P. =a +q(a* -a@). 
By setting P; = a, the above equation can be solved for q: 


a-a 
q= . (10.6) 


a*-a” 


c. The likelihood ratio is the same as the Bayes likelihood ratio in the case of simple 
hypotheses. 
d. Unlike the BLRT, the threshold n is only determined by a value a. 


e. If T = T(x) is statistically sufficient for 0, the likelihood ratio test (LRT) depends on 
x, only via T(x). Indeed, if f(x;0) = g(T,A)h(x), then: 


_ g(T,6;) _ 
Aa)= Seg y = MD) 


Conclusion: We can formulate LRT based on the PDF of T, instead of using the PDF of the 
entire x data sample. 
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10.5 ROC Curves for Threshold Testing 


All threshold tests have P, and P,, which are parameterized by a value 7. The receiver 
operating characteristic (ROC) curve is simply the depiction of the parametric curve 
{P-7,9),Pp(1.Mhnq Equivalently, ROC curve is the depiction of # = Pp with respect to a = P; 
(Figure 10.19). 


Properties of the ROC curve: 


1. The ROC curve for two equiprobable states, e.g., the coin flipping (g(x) = q, inde- 
pendent sample data) is a diagonal straight line with a slope = 1. 


a=P, =E, [9]=4, 
B=Pp =E, [9] =9. 


2. The ROC curve of an MP test is always located over the diagonal: The MP test is 
objective (a test is objective if the probability of detecting / is at least equal to the 
probability of a false alarm a:f = a). 


3. The ROC curve of each MP test is always concave. 
Figures 10.20 through 10.22 illustrate these properties. 


To determine the concave behavior of the ROC curve, let (a,,f,) be the level and the 
power of a test y,, and (@,,/,) be the level and the power of a test y2. We define the test: 


Py =P, +(1-p) qo. 


ROC 


FIGURE 10.19 
The ROC curve. 
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FIGURE 10.20 
Illustration of the ROC curve for equiprobable detection. 
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FIGURE 10.21 
Plot of the ROC curve, which is always located above the diagonal for MP tests. 
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FIGURE 10.22 
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The ROC curve for each MP test is always concave. A non-smooth curvature test (bold line) can be improved by 
randomization, through which two reference points (a,,,) and (a,,/,) are linked by a straight line. 


This test can be applied by random choice of y, and gy, with probability p and 
1-p, respectively. The level of this test is expressed as: 


Ay = 


‘ol P12] =p 


ioL9,]+(1— p)Eolp2]= pa, +(1— p) a, 


and its power reads as: 


Bi = 


By [9.2] = p B, +(1— p) Ba. 


Therefore, as p varies between 0 and 1, gy, has a performance (45, #,,) that varies 
on a straight line connecting the points a, 2, and a,, fp). 


4. If the ROC curve is differentiable, the threshold MP-LRT is necessary to approxi- 
mate the minimax P., which can be calculated graphically via the ROC gradient at a: 


d 
= dao 


5. When the hypotheses H, and H, are simple, the MP-LRT threshold necessary to 
approximate the minimax P, can be calculated graphically by the intersection of 
the line P,, = 1-P, = P; with the ROC curve. 


The above properties are illustrated in Figures 10.23 and 10.24. 
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a Desired value 1 


FIGURE 10.23 
The MP-LRT threshold can be calculated by differentiating the ROC curve. 


Minmax a 1 


FIGURE 10.24 
The threshold of the minimax Bayes test, which can be calculated by the appropriate line intersection. 
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Example 30: Test with Respect to an Identical PDF 
Let two hypotheses for the scalar RV x: 


Ay: f(x) = fo(x), 
A: f(x)= fi), 


where f, and f; represent two PDFs, as illustrated in Figure 10.25. 
Objective: Derive the MP-LRT. 


SOLUTION: 


The LRT is expressed as: 


_ f 1(X) . 
A(x) rae < n, 


or, alternatively: 
Ay 
fix) : 1 fo(x). 


Ao 


In Figure 10.26, it is obvious that for a conditional n, the decision area H, is: 


/4<x<1-n/4;, 0<n<2 
_}t{n n/4}, O<n 


a 
©, otherwise 


In order to choose the appropriate threshold n, we must consider the constraint P; = a. 


fo(x) (x) 


Zz 


1 X 1/2 1 x 


FIGURE 10.25 
Two PDFs under test. 
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FIGURE 10.26 


The region ¥, for which MP-LRT decides on the hypothesis H, and is a set of x values for which the triangle 
exceeds the horizontal line of height . 


At first, we try to set a value to 7 without randomization (g = 0). Therefore, we consider 
1 € [0,2]: 


1-7/4 
a=P(xe%|Ha)= J flxddx 


n/4& 


=1-7/2. 
Thus, by solving with respect to 1, it yields: 
n= 2(1-a), 


and, therefore, no randomization is required. 
The MP-LRT power (probability of detection) is expressed as: 


1-n/4 


Py =P(xeX,|H,)= J fi(x)dx 
n/4 
1/2 1/2 
=2) fcjdr=2 f ardx 
n/4 n/4 


=1-1°/4. 


By substituting 7 = 2(1 — a) in the above relation, the ROC curve is derived: 


B=1-(1-ay. 
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Example 31: Detection of Increase in Poisson Rate 


Let x be the readout of the number of photons collected by a charge array in a certain 
period of time. The average number of photons @, that are falling upon the array is 
constant and known because of the environmental conditions. When there is a known 
photon source; the photon rate is increased to a known value 6,, where 6, > 6). The objec- 
tive of the photodetector is to detect the presence of activity based on the measurement 
of x. Typically, the RV x is Poisson-distributed: 


x~ fia) = Ze, x=0,1,... 
x! 


and the problem lies in detecting the change from 6, in 6,, with respect to the 6 Poisson 
rate parameter, i.e., in modeling of the test using the following simple hypotheses: 


Hy :0=6), 
H,:0=6,, 


where 6, > 0) > 0. At this point, we want to design the MP test at a predetermined level 
a € [0,1]. 


SOLUTION: 
We already know that the MP test is modeled as the LRT: 


Ay 


ny=( 2] ei > 1). 


Ho 


Since the function of the logarithm is a monotonically increasing function and 6, > 45, 
the MP-LRT is equivalent to the linear test: 

Since the Poisson CDF is not continuous, only a discrete set of values is reachable by 
the normalized LRT test: 


where: 


_ Inn+6,-4 
In(@,/ 8)” 


First, we shall try to set the threshold y without randomization: 


a= Py (x>7)=1- Po, (7), 
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F(x|H,)=P(X = x|H) 


e 
1-a 8 
1-a itd cocccer ooo od 


l. 
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FIGURE 10.27 
The CDF of the LR test for the increase of the Poisson rate, which is presented as a staircase function. 


where Pog, (©) is a CDF of a Poisson RV with 0 rate. In this case, we encounter a dif- 
ficulty, which i is graphically described in Figure 10.27. 

As the Poisson CDF is not continuous, only a discrete set of values is realizable by the 
normalized LRT test: 


ae {aj}, a;,=1—Po, (i). 


Let a € (4,,4;,,). In this case, we must randomize the LRT test with the choice of y, q so 
that the following relation is satisfied: 


a=P,(x>7)+q P(x =). 
Following the aforementioned procedure and using Relation 10.6, we choose: 
7 =7" = Pog, (1—4;), 


which yields: P,, (x > y*) =a, We set the random parameter according to: 


Based on the above relations, the probability of detection is obtained: 
Po=Py(x>y*)+q* Py (x=), 


which is illustrated in Figure 10.28. 
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A=10 


FIGURE 10.28 
The smooth curve expresses the normalized MP test, while the staircase curve is the non-normalized LRT test. 


Example 32: On-off Keying Technique (OOK) to Gaussian Noise 


On-off keying (OOK) is a type of binary modulation used in many digital and optical 
communication systems. In the time interval for the transmission of one digit, the built- 
in output x of the receiver can be modulated as either only noise w (if the transmitted 
information is zero) or a constant, assumed to be 1, plus the additive noise. The detector 
must decide between the following hypotheses: 


Hy) :x=w, 
H,:x=1+u, 


where it is true that w ~ N(0,1), ie., the received SNR (Signal-to-Noise Ratio) is OdB. 
The LR test is expressed as: 


: e- ae -1y ‘ 
V2n 2 aay 
A(x) = i ; =e 
e-— x 


The ROC curve is derived, as usual, by the equation: 


P, = P(x >4|H,) = P(x-1>4-1]H,)=1-N(A-1) 
© B= Py =1-N(A-1)=1-N(N7(1-a)-1). 
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FIGURE 10.29 
The ROC curve for the OOK Gaussian example. 


This curve is illustrated in Figure 10.29 for three different modeling approaches of the 
LRT threshold: LRT Bayes, minimax LRT and MP-LRT. 


1. Bayes test for the minimization of P,, considering equiprobable hypotheses 
Ay Ay(y = 1): 


2. Minimax test: 
x 7 Inn+ 1a e 
< 2 
Ho 
where the parameter A is chosen to satisfy the condition: 
P, =1—-N(A)=N(A-1)= Py. 


The solution to this equation is, again, \ = 1/2, since NM(—x) =1-NV(x). 
3. MP test at level a: 
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TABLE 10.1 


A Typical Performance Comparison for Three Different Threshold 
Settings in OOk Gaussian Example 


P, Pp P. 
Bayes 0.31 0.69 0.31 
Minimax 0.31 0.69 0.31 
MP 0.001 0.092 0.5 


where a= P(x >A|H,))=1-N (A), or, alternatively: 
A= N(1-a). 


A typical performance comparison is given in Table 10.1, where it is determined 
that the false alarm level is a = 0.001 for MP-LRT (the corresponding MP-LRT 
threshold is 4 = 2.329). 


We can see from the above table that the Bayes and minimax tests perform 
the same since they use the same threshold. On the other hand, the MP test has a 
much lower error rate but also presents a significantly smaller probability of detec- 
tion, Pp, and a greater error probability, P,. 


Appendix I: Introduction to Matrix Algebra 
and Application to Signals and System 


Many computational engineering applications require the derivation of a signal or a 
parameter of interest from degraded measurements. To achieve this, it is often useful to 
deploy fine-grained statistical models; diverse sensors which acquire extra spatial, tempo- 
ral, or polarization information; or, multidimensional signal representations, e.g., in time 
scale and time-frequency scale. When applied in combination, these approaches can be 
used to develop very sensitive algorithms for signal estimation, detection and monitoring 
that can utilize the small but lasting differences between signals, interference and noise. 
Additionally, these approaches can be used to develop algorithms to identify a channel or 
system that generates a signal under the influence of additive noise and interference, even 
when the channel input is unknown but has known statistical properties. 

Generally, statistical signal processing involves the reliable estimation, detection and 
classification of signals subject to random variations. Statistical signal processing is based 
on probability theory, mathematical statistical analysis and, recently, system theory and 
statistical communication theory. The practice of statistical processing of signals includes: 


1. Description of a mathematical and statistical model for measurement data, includ- 
ing sensor, signal and noise models. 


2. Careful statistical analysis of the fundamental limitations of the data including 
deriving benchmarks on performance, e.g., the Cramer-Rao, Ziv-Zakai, Barankin, 
Rate Distortion, Chernov, or other lower bounds on average estimator/detector 
error. 


3. Development of mathematics of optimal or sub-optimal estimation/detection 
algorithms. 

4. Asymptotic analysis of error performance, which derives that the proposed algo- 
rithm is approaching a benchmark resulting from (2). 

5. Simulations or experiments that compare the performance of the algorithm with 
the lower limit and with other competing algorithms. 


Depending on the application, the algorithm may also be adaptive to variations in signal 
and noise conditions. This requires the application of flexible statistical models, of low 
complexity real-time estimation and filtering algorithms, as well as online performance 
monitoring. 
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as 
A.I.1 Basic Principles of Vectors/Matrices 
A.1.1.1_ Row and Column Vectors 


A vector is a list of n sorted values: 


In the case where the data x; = u + jv are complex numbers, the Hermitian transpose 
(conjugate-transpose) is defined as follows: 


where x; = u—jv is the conjugate of x,. 

Some vectors that are often encountered below are the vectors containing only ones and 
the elementary j-vector, which constitutes the j-th column of the unitary matrix (identity 
matrix): 


1=[1L-,1], a= Os 1 acl 


yi 
j-th element 


A.L1.2 Vector Multiplication 


The inner product of two vectors x and y with an equal number of elements 1 is calculated by: 


n 
Drei 
xy = > ivi 
i=1 


The norm-2 Ix, of the vector x equals: 


|x|= Vx'x = 
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The outer product of two vectors x and y with an equal or unequal number of elements, 
n,m, yields a matrix of n x m dimensions: 


xy" =(iyiijn 
= [xyueX¥m | 


XY ass XY in 


XY 7 XY m 


A.I.2 Orthogonal Vectors 


If xy = 0 is true, then the vectors x and y are said to be orthogonal. Moreover, if they are 
both of unit lengths, x|=1 and ly| = 1, then they are called orthonormal vectors. 


A.I.2.1 Vector Matrix Multiplication 


Let A be a matrix of m X n dimensions with columns 4.,,---,4., and x be a vector with n 
elements. The product Ax forms a column vector, which is derived from linear combina- 
tions of the columns of A 


Let y be a vector with m elements. The product y“A is a row vector resulting from linear 
combinations of the rows of A 


y A= S vite 
i=1 


A.1.2.2 The Linear Span of a Set of Vectors 


Let x, ... ,X,, be p-element column vectors, which form the p x n matrix 


X=[x,,...,X, ]- 
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Let a=[a,,...,a,]'.Then, y= yiax, = Xa is a p-element vector, which is a linear combi- 
i=1 
nation of the columns of X. The linear span of vectors x,, ... ,X,, is defined as the subspace 
in the R? domain including all the possible linear combinations 


span{x,,....x,}={y:y=Xa, aeR"l. 


Therefore, when vector a extends to the entire R? domain, vector y extends to the linear 
span of x1, ... ,X,.- 


A.1.2.3 Rank of a Matrix 


The rank (of the columns) of a matrix A equals the number of its linearly independent col- 
umns. The dimension of the column space of a matrix A of rank p is equal to p. 
If matrix A is of a full rank, then: 


0=Ax= )' xa, ox=0. 


Additionally, if A is a square matrix, then it is nonsingular. 


A.1.2.4 Inverse Matrix 


If A is a square nonsingular matrix and A‘ is its inverse, then AA™ = I. In the specific case 
of a 2 x 2 matrix, the inverse matrix is given by the Cramer equation: 


-1 

a b 1 d —b 
— y ad be. 
: | an 4 *] er 


In some cases where a matrix has a specific type of structure, its inverse can be expressed 
in a general way. 
A typical example is the Sherman-Morrison-Woodbury property: 


-1 


[a+uv'] =A7t-a“uli+Vv'A7u] v7A7, (A.1) 


which assumes that: 


e A,U, V are compatible matrices (of appropriate dimensions to allow the multipli- 
cation between them) 


e [A+ UV" and A* exist. 
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Also, a useful expression is the partitioned matrix inverse identity: 


aa —1 
Ay Ay [Au = A,AZAs | -ATAy [ Ass - A,ATA) | 


A» Ay 


-1 —1 
-ADAn [Au ~ A AZA,, | [ A» = A, ATA | 
(A.2) 


which assumes that all the inverse submatrices exist. 


A.1.2.5 Orthogonal and Unitary Matrices 


A square matrix A with real number elements is orthogonal if all of its columns are 
orthogonal: 


ATA=L (A.3) 


The condition for the generalization of the concept of orthogonality for complex matrices 
is that A must be unitary: 


AMA=L 
Relation A.3 indicates that if A is orthogonal, then it is also invertible, so that: 


AT=Al’. 


A.1.2.6 Gramm-Schmidt Orthogonality Method 


Let x,, ...,x,, be the column vectors of p linearly independent elements (1 < p), whose linear 
span belongs to the subspace H. The Gramm-Schmidt method is an algorithm applied to 
these vectors in order to create a new set of 1 vectors yj, ..., y,, which are produced in the 
same subspace. The algorithm is implemented as follows: 


Step 1: Random choice of y, as an arbitrary starting point in H. For example, let 
r 
a,= [aii | , Y, =Xa,, where X = [x,, ..., X,]- 


Step 2: Derivation the rest of n-1 vectors y,, ... y,, through the following iterative 
procedure: 


j 
For j=2 ny) =x)— Y Kyi, where Kj=x}yja/YjaYj- 


i=1 
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The above procedure can equally be expressed in the form of matrices as: 


Y=XH, 


where Y = [y,, ---, y,,], and H is the Gramm-Schmidt matrix. 

If, in every step j = 1, ---, n of the process, the length y; is normalized, that is, 
y,;-Y,;=y;/ ly the algorithm produces orthonormal vectors. This is called Gramm- 
Schmidt normalization and produces a matrix Y with orthonormal columns and identi- 
cal column span as that of X. This method is often used to create an orthonormal basis 
[Vas ese Al in the R? domain, arbitrarily starting from a vector y,. The matrix that is derived 
is of the following form: 


val 
y=-|“? 
Vi 
and 
Y'Y=L 


where v;, «+, v,, are orthonormal vectors that achieve completion of the basis with respect 
to the original vector y;. 


A.1.2.7 Eigenvalues of a Symmetric Matrix 


If the n X n matrix R is symmetric, that is, if R7 = R is true, then there is a set of 1 ortho- 
normal eigenvectors v;: 


i os3 
Ty =A.= rd 


a 0, i#j 
as well as a set of the corresponding eigenvalues, so that: 


Rv, =A,v;, i=1,--+,n. 


The above eigenvectors and eigenvalues satisfy the following conditions: 


vi Rv; = Ki 


viRv;=0, i#j. 
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A.1.2.8 Diagonalization and Eigen-Decomposition of a Matrix 


Let the 1 X 1 matrix U = [v,, ---, v,], which has been derived from the eigenvectors of a 
symmetric matrix R. If matrix R is a real-symmetric matrix, then the matrix U is real- 
orthogonal. In the case that the matrix R is complex-symmetric, then the matrix U is 
complex-unitary: 


U'U=IL, (U: orthogonal matrix) 
U"U=I, (U:unitary matrix). 


Since the Hermitian conjugate-transpose operator is general, next it will be used to 
denote an either real or a complex matrix (A” = A’, for real matrix A). 
The matrix U can be used for the diagonalization of R: 


U"RU= A. (A.4) 


In cases of real and Hermitian symmetric R, matrix A is diagonal with real values: 


hye + 
A=diag(Aj=| i. |, 
0 rn 
where A; depict the eigenvalues of R. 
Expression A.4 suggests that: 
R=UAU", 


which is called the decomposition of matrix R. Since the matrix A is diagonal, the above 
self-decomposition can be expressed as a sum of terms: 


R= aviv (A.5) 
i=l 


A.1.2.9 Square Form of Non-Negative Definite Matrix 


For a square-symmetric matrix R and a compatible vector x, the square form is a scalar 
coefficient defined as x’Rx. The matrix R is non-negative definite if for each x it is true that: 


x'Rx>0. (A.6) 
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Matrix R is positive definite if the equality is excluded from Relation A.6, so that: 


x'Rx>0, x#0. (A.7) 


Examples of non-negative definite matrices: 


¢ R=B'B, for any matrix B. 


¢ Symmetric matrix R only with non-negative (positive) eigenvalues. 


Rayleigh Theorem: 
If ann Xn matrix A is non-negative definite with eigenvalues {ri}, it is true that: 


wAu 


min(A;) < —; 
uu 


<max(A,), 


where the lower limit is encountered when u is an eigenvector of A associated with the 
minimum eigenvalue of A, while the upper limit is encountered by the eigenvector associ- 
ated with the maximum eigenvalue of A. 


A.I.3 Positive Definiteness of Symmetric Partitioned Matrices 

If a matrix A is symmetric and can be partitioned as described by Relation A.2, then it is 
true that: 

-1 
Ay Ay 
Aa Ax 


Ay -ApApA, 0° 
0 A» 


A= 


_ I -A,Az 


-1 
I 0” 
0 I -A>A, I 


(A.8) 


as long as the submatrix A3; exists. Therefore, if the matrix A is positive definite, then 


the submatrices A,,—Aj,,A>,A,, and A, are positive definite. Similarly, the submatrices 
A» —-AyAjA,» and A, are positive definite. 


A.1.3.1 Determinant of a Matrix 


Let A represent a square matrix. Its determinant is then defined as: 


l=] J. 


NOTE: A square matrix is non-singular if and only if its determinant is non-zero. 
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If the matrix A can be partitioned as in Relation A.2, and if matrices Aj! and Aj; exist, 
then: 
\A| = |An|A2 — A AA = |Ax||An = AA An ’ 


which is derived from Relation A.8. 


A.1.3.2 Trace of a Matrix 


For any square matrix A, the trace is defined as: 


trace{A} = Sai => iG 


The following properties are true: 


° trace{ AB} = trace{BA}. 
© x'Rx = trace{xx'R}. 


A.1.3.3 Differentiation of Vectors 


Differentiating functions containing vectors is very common in signal detection and pro- 
cessing theory. 

Let h= [ hh, T be ann X 1 vector and g(h) be a scalar function. Then the gradient 
of g(h), which is noted as Vg(h) or V,,9(h), is defined as the column vector of the following 
partial derivatives: 


dg(h) deh) | 
[A AT. 


Moreover, by introducing a constant, c, it is: 
le xe) 3 
V,,(h’x) = V,(x"h) =x, 
and if Bann Xn matrix: 
V,,(h—x)! B(h—x) = 2B(h—x). 


For a vector function g(h) = [g,(h), «+, g,,(h)]’, the gradient of g(h) is an m x n matrix. 
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Appendix II: Solved Problems 
in Statistical Signal Processing 


A.II.1_ Compute the CDF F,(a), the expected value E[x], the second statistical moment 
i[x*] and the variance o? of a RV, which has the following PDF: 


ee for 0<a<2, 


f(a)= a" 


0, otherwise. 


SOLUTION: The following graph illustrates the given PDF. 


SAa) 
1 


The CDF is equal to: 


oo 


2 
[0K Jauee©, O<a<2. 
2 4 


—co 


Fea)= f felu)du= 


The expected value of the RV x is expressed as: 
+00 


i[x] = fo. )aa= =. 


The second statistical moment of the RV x is equal to: 


+00 too 


[x2] = Je? flayda = fe(1-S)a- =. 


co co 
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The variance can be easily calculated from the above results: 


of = E[x"]-(Elx}), 3G) “9 


Moreover, from the CDF, we could easily calculate the probability of the RV x 


being within a range of values. For example, the probability 0.5 < x < 1.5 is calcu- 
lated as: 


P[O.5<x<1.5]=F,(1.5)—F,(0.5) 
iS 2 4 


"16 16 2° 
This value corresponds to the shaded area of the figure. 


A.II.2 Calculate the expected value and the variance of an RV x, the PDF of which is 
uniform and is illustrated in the following figure: 


Sa) 


0.5 


SOLUTION: 


too 


3 2 
[x] = fof @rda = \o6ades05 
1 


3 


ee 
1, O\2 9 


i He ae 


2 


oo 


foo 


3 
3 
[x2] = fe f.a)da= {0.5a2da=0.5" 
1 


3) 243 3 6 30 
Thus: 
ge ge teow 1S ae 
= iy _ Yy = 2 = = 
o? = B[x”]-(E[x]) - aca 


A.II.3 Let the joint PDF of two RV x, y be: 


A, for -l<a<l, O<b<1, 
Fy(a,b) = 


0, otherwise. 
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Derive the value of constant A and calculate the probability of x and y being in 
the interval 0.5 < x <1, 0 < y <1, respectively. 


SOLUTION: From the basic property of the joint PDF, we have: 


ja 


The probability that x and y will be in the interval asked is illustrated in the 
shaded area of the following figure: 


foo oo 


J J fe y(a,b)dadb =1 


ja 


ol [dea= A 


-10 


@&2A=1. 


= Aalb) =1.A-2-1=1 


Thus, A = 1/2. 


tt 4 
Plos<x<10<y<1 af fa ie 
[0.5<x ys1]= ce ae ae 


0 0.5 


b 


-1 005 1 4 


A.IL.4 Let the random sinusoidal signal x(n) =A cos(@pn + g), at the n point in time 
with a PDF for A and @: 


1 

—, O<as4, —, O<qg<2z, 
fala)=44 fo) = ? 

0 


otherwise, 0, 


Calculate its statistical properties. 


SOLUTION: The RVs A, g are statistically independent (the amplitude and phase of a 


signal fluctuate independently), thus their joint PDF is equal to the product of the 
individual PDFs: 


1 
—, Osas4, 0<Q@<2z, 
fao(@ ?) = 482 ? 


0, otherwise. 
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The expected value of the signal is expressed as: 


E[x(1)] = 


ai acos(@yn + yp) dado 
0 


0 


= i fos] eal lowe 


= “[sin(ooyn+ 2m) — sin(@yn) |= 0. 


The second statistical moment of the signal is: 


2n 


4 
i[x*(n)] = —| a’ cos*(@n +p) dado 


= 2 fe il foe ome} 


The variance of the signal is equal to the second statistical moment since the 
expected value is 0: 


o? = E[x’]-(E[x]) =E[x?]-0=—. 


The autocorrelation function is: 


Ry(mm,n) = EL x(m)x"(n)| 
= at a a } cos(@)I1 + ~) cos(@n+ g) do 


0 


= 5 cos(o(m —n)). 


As shown in the following exercises, this sinusoidal signal is stationary in the 
wide sense, since it has a constant mean value ( i[x(n)] = 0), and the autocorrela- 
tion function depends only on the difference of the m and n points in time (since it 
is true that R,.,(m,n) = (8/3) cos (@)(m-n))). 
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A.IL5 Let: 


x(n) = Acos@n+ Bsinan, 
y(n) = Bcosa@n— Asinan, 


where @ is a constant and A, B are the independent RVs of zero expected value and 
of variance o?. Calculate the cross-correlation of x() and y(n). 


SOLUTION: 


R,,(m,n) = E[x(m)y"(n)] 


= E[(A cos@m-+ Bsin@m)(Bcosan- Asinan)| 


= 


= E|ABcos@m-cosa@n— A” cos@m:-sinan 


+B’ sina@m-cosa@n—ABsinem-: sin on| 


= E[(AB)|cos@m-cos@n-—E[A?]cos@m-sinan 


+ 


E[B*]sinam-cos @n—E[AB]sinam-sinan. 


However, because of the independence of the RVs, it is true that: 


[AB] = E[A]-E[B]=0. 


Also, it is given that: 


)[A?] = E[B?] = 0°. 


According to the above relationships, the cross-correlation function becomes: 


R,,(m,n) = o’(sin@m-cos@n—cosa@m:sinan) 


=o” sin(m—n). 


A.II.6 Calculate the expected value and the autocorrelation of the discrete-time ran- 
dom sequence x(n) = A", n > 0, where A is an RV with identical distribution in 
the interval (0, 1). 


SOLUTION: 


1 O<a<il, 
0, otherwise. 


fo=| 
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i gq} 
®[x(n)] = ELA"] far da 
0 


at = 
n+m+1- 


1 
R,,(n, m) = a[ x(x" (m)] = [A"A™] —f- [an | _ fern da= 
0 


A.IL7 Prove that: R,,(n, n) > 0 


SOLUTION: 


Rea(nt,n)=BLx(n)x"(n) = BL xt) | 0. 


A.IIL.8 Let arandom sequence x(n) = An + B, where A, B are independent Gaussian RVs 
with zero expected value and variances On oe, respectively. Calculate: 
a. The expected value and the autocorrelation of the random sequence x(n). 
b. The second statistical moment of the random sequence x(n). 


SOLUTION: 
a. 


i[x(n)] = E[An+ B] = E[A]n+ E[B] = 0, since E[A] = E[B]=0. 


R,.(n, m) = E| x(n)x*(m) |= E[(An+ B)(Am+ B) | 
= E[A2nm+ ABn+BAm+B?| 
= E[A’]nm+ E[AB]n+ E[BA]m+ E[B’] 
=o7.nm+ E[AJE[B]n+E[B]E[A]m+ o; 


since E[AB]= E[A]- E[B]=0, because of the independence of the given RVs. 


b. Following the same way of thinking as in the last relation and substituting 
m —> n, itis derived: 


)[x?(n)] = E| x(n)x"(n) ]=---= 03? +03. 


A.II.9 Let the random signals x(n) = A cos(wn + 0), y(n) = A sin(wn + @), where A, w 
are constants and @ is an RV with identical distribution in the interval [0, 27]. 
Calculate their cross-correlation: 
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SOLUTION: 


R,(n+1,n)= B| x(n+Dy*(n) |= E[Acos(o(n +1) + 6)- Asin(on+6) | 
= A°E| cos(w(n+/)+ 6)-sin(on+ 6) | 


7 ah i sin(—col) + sin(2on + ol + 26) | 


2 


= [-sinol+ s[sin(2on + ol + 26) |]. 


Further analysis of the second part of the last equation yields: 


s[ sin(2e@n + al +26) |= } sin(2on + wl +26) f (6) d0 


2) 
= | sin(2on+ ol +20) >-d6 
, 20 


2a 


=f 
On 2 
0 


sin( 2oon-+ ol+ 26)926 


2n 
=~ i sin(a+26)d(a+26) 
4n 
0 
=- eae + 20)" =0. 
4 0 
Consequently, the first relation is now written as: 
A2 
R,,(n+1,n)= gS = R,,(). 


Next, the cross-correlation R,,(n + I, n) is calculated: 


yx 


R,,(n+1,n)=E| y(n+)x*(n) |= E[ Asin(o(n +1) +)- Acos(on+ 6) | 


= A*B| sin(@(n+1) + @)-cos(@n+6) | 


= ae | sin wl + sin(2on + ol + 26) | 


2 
= [snes di aaaz2)| 


=0 


A’, 
= ol = R,, (1) =—R,, (2) = R,y(—D. 


Thus, we have proven that: R,,(/) = R,(—/). 
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A.II.10 Let the random signal x(n) = A cos(wn + 6), where 0, w are constants and A is an 
RV. Determine if the signal is wide-sense stationary (WSS). 


SOLUTION: In order to determine whether the signal is WSS, we should examine 
whether its expected value is stationary and whether its autocorrelation is inde- 
pendent of time but depends only on the time difference m — n = 1. 


i[x(n)] = E[A cos(@n + 8)] 
=cos(an+ @)E[A]. 


We can see that the expected value is not constant, unless E[A]= 0. 


R,,(m,n) = E[x(m)x"(n)] = E| Acos(am +6). A cos(an + 8) | 


= E[A?]cos(@m + 4) cos(@n + @) 


= 1A") 5[cos(o(n— n)) + cos(w(m+ n) +20) |. 


We note that the autocorrelation is not dependent only on the difference m — n 
and, therefore, the random signal is not WSS. 


A.II.11 Let x(n) be a WSS signal and R,,(/), its autocorrelation. Prove that: 


BL (x(n +1)—x(n))? |=2[R,(0)-R,.() ]. 


SOLUTION: 


| (x0n41)—x(n)) | =E| x?(n+1)-2x(n+)x(n)+ x°(n) | 


= E[ x?(n+1) |- 2E[ x(n +1)x(n) ]+ El x2(n) | 
= R,,(0)—2R,,() + R,,(0) 


= 2] R,.(0)-R,.()) |. 


A.II.12 Let x(n), y(n) be two independent and random WSS signals. Determine if their 
sum is WSS. 


SOLUTION: We define s(n) = x(n) + y(n). 


a[s(n)] = B[x(1) + y(n) = B[x(1)] + E[y(1)].- 


Since the signals are WSS, their expected value will be constant. Therefore, 
their sum E[x(n)]+ E[y(n)] will also be constant. 
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The autocorrelation is calculated as follows: 


R,,(n+1,n) = E[s(n+ 1)s°(n)] 

= E[ (x(n) + y(nt Dy(x(n)+ y(n) | 

= E[ x(n +)x(n) + x(n+Dy(n)+ y(ntDx(n)+ y(n+l) y(n) | 
= B[x(n+)x(n)]+ Elx(n+Dy(n)]+ Ely(n+)x(n)] 
a[y(n+l) y(n) 

= R,() + E[x(n + DIE[y(n)] + Ely(n + DIJE[x(7)] + R,, (2) 
=R,.()+R,, (+ 2E[x(JE[y(1)]. 


+ 


Thus, the signal s(n) is WSS since its expected value is constant and its autocor- 
relation depends on the time difference, /. 


A.II.13 Calculate the expected value and the variance of a WSS signal, the autocorrela- 
tion function of which is: 


104017" 
1437 


R,.()= 


SOLUTION : It is true that lim, ,,.. R,.(J) = (E[x(n)])”. Thus: 


= +21 1 
(B[x(n)])° = lim,.,,.. R,x(2) = lim,.,... = a 
z +3 


Therefore, itis E[x(1)]= +V7 : 


I[x7(n)] = R,(0) yields E[x*(n)] = 10. 


Finally, the variance is calculated from o? = R,,(0) -( aLx(n)]) =10-7 =3. 


A.II.14 Let x(n) be an identical distributed random signal, the PDF of which is illus- 
trated in the figure below. Calculate the expected value and the variance of 
the output signal y(n) when the input x(n) is applied to a 3-point casual moving 
average filter with the impulse response h(n) = {1/3, 1/3, 1/3}. 


f(a) 
0.5 
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SOLUTION: For the given input signal, it is already known from a previous exercise 
that E[x]=2, 02 =1/3 Vn. Thus: 


A : 111 
By] = Ex] > (k) = ELx](I(0) + h(1) + (2) = ots ; +h)n2 


The variation of the output signal is: 


03 =o; >, |h(k)) = 02 (h?(0) + H°(1)-+ h°(2)) = a oe : 5 


k=-co 


A.IL.15 A random WSS signal with expected value 1 — e* is applied to a linear time- 
invariant (LTT) system, the impulse response of which is: 


h(n) =3e°"u(n). 


Calculate the output. 
SOLUTION: 
sly] = Lx] ” n(k) = B[x]H(e”). 
k=-00 
But: 


H(e!”)= ¥ ner" = >: -2n 9 jon 


n=—c0o 


2 =3¢ -Q+jo)n _ 1 
307" 


n=0 


For w = 0, it is: 


H(e”)=3 


Finally, the expected value of the output is: 


iyl=BitH(e")=(1-e*} 9a 
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A.II.16 Let a stochastic signal have an expected value, E[x]=2, and variance, 
o a This signal is applied to a casual filter with transfer function 


H(z)=- @ a+52 zo i z*. Calculate the expected value and the variance of the 
output signal. 
SOLUTION: 
Z 
H(z)= it ; a yet > h(n) = 50) + ; 6(n-1) 50 2) 


Ly] = Bk] n(k) = Bx H(0) + h(1) + h(2) f-1411]-2.0=0 
k=—00 
G,=0; 25 [MP = 02[|h(O)P +| AA) P +] (2) | 
k=—00 
12 1 ' 12 3 
+—+ = = 
4116 4 16] 48 32 


A.I1.17 A white noise signal x(n) of zero expected value and variance 0% is applied to 
an LII system with impulse response /i(1) = (0.6)"u(n) producing the output o(7). 
The signal v(1) is then applied to a second LTI system of impulse response 9(n) = 
(0.8)"u(n), producing the output y(n). Calculate the expected values and variances 
of the signals v(n) and y(n). 


SOLUTION: For the first system with h(n) = (0.6)"u(n), it is: 


i[v] = > h(n) = E[x]H(e’”) = 0, since E[x] = 0. 


o=0° >> lac) =o. = } (ey) do 


n=—co 


by using Parseval’s theorem 


1 
do 
1- 7  1-0.6¢ce 


|e He) do= 02 za 


21 1 2 1 20 
=O. dw =o; 
2m J 1.36—1.2cos@ 20 (1.36) (37 
1 Co 
=0, = —_*_=1,50250., 


"0.64 (0.8) 
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where the fourth equality is due to the fact that: 


H(e”) = DTFT{x(n)} = DTFT{0.6" u(n)} = 1/(1-0.6e"”), 


while the last integral is solved by the use of the relation: 


2a 


} do 20 


a+boos@ ./,2_ 72° 
a’ —b 


For the second system with g(1) = (0.8)"u(n), it is: 


ily] = 10> g(n)= u[v]G(e’°) = 0, since E[v] = 0. 


n=—co 


o2=02) Iain) = 025 [cee do 


n=—0o 


= [ct™ce)da = 03 u } ee 
i 2n J 1-0.8e” 1-0.8e" 


9 1 20 


pe { 1 
=0; d@=0; 
2n J 1.64-1.6cos@ 2m .|(1.64)? -(-1.6) 


oy 


2 2 10) Q)\2. 2 
GoGo _ MOBY __O2 _ _ 4.345, 
0.36 (0.6)? (0.6)? (0.48) 


A.II.18 A WSS white noise signal x(n) of zero expected value is applied to a casual LTI 
discrete-time system with an impulse response h(n) = (1) — a5(n — 1) producing 
the WSS output o(). Calculate the power spectrum and the average power of 
the output y(7). What is the effect of a on the average power of the output? 


You are given the following formulas for the calculations. 
¢ Power spectrum of the output: P,,(@)=|H (ce)? P..(@). 


¢ Autocorrelation of white random process WSS: P,,(@) =| H (ec)? P..(@). 


SOLUTION: It is true that: 


H(e!”) = DTFT{h(n)} = y h(n)!" =1—ae"!”. 


n=—00 
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Thus, 


P,,(@)=|H(e®)| Po) 
= H(e)H(e-)P.,(@) 
=(1—ae!)(1-ae”)o? 


=(1—ae! —ae” +a7)o?. 


The average power for a WSS signal is given by: 


P, =R,,(0)= 0, +(E[y])’. 


But: 


sly] =Blx] * h(n) =0, 


n=—0o 


oo un 
1 jo 
1-0; D)IMoP=o3 5, [I H(e™)P deo 


by using Parseval’s theorem 


nu 7 a 
=o H(e”) do = 02 = | (1+a—2acose)de 
20 “20 
—t 


1 


roe on] ints] moo 
20 


1 —1 


x 


=o? 1 fasa)20- 2a-0]=02(1+a7). 
“20 
Alternatively, the variance can be calculated as follows: 


02 =o? } | h(n)? =02(|M(O)P +|n) P)= 021 +0"). 


n= 


We can see that the average power is increased with the increase by a. 


A.II.19 The input x(n) to the discrete-time system of the figure is white noise with 
power spectrum o?, ie., P.,(@) = 0”. Calculate the power spectrum and the aver- 
age power of the output y(n). It is given that a < 1. 
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SOLUTION: The power spectrum of the output is P,,(@)=|H (e!”) ? P..(@). From the 
figure, it is y(n) = x(n) + ay(n — 1) and, taking the DTFT of the two parts of this 
relation, it is derived: 


Y(e”) = X(e”) +. ae” Y(e!”) 
Y(e"”) mee 


= Fi(giya—  _, 
X(e’”) 1-ae!® 


a<1, |oa|<z. 


Thus: 


P,,(@) = H(e””)H(e")P,,.(@) 
4 — 
~ Jae 1—ae 

Co 


1+a°-2acosa’ 


lol <x. 


The average power of the output y(1) is given by: 


al | y(n) P |= Ryy (0). 


The autocorrelation R,,(/) is derived by the inverse Fourier transform of the 
power spectrum P,,,(@): 


jool 
R,,(1) = IET{P,y( ak Moye deo 


2 2 2 
we } y el! da =—- a! 
2n J 1+a°—2acos@ 1-a’ 


Finally, for / = 0, it is: 


Ry(0)= 3 = B[| yf]. 
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A.II.20 A WSS white noise signal x(n) with zero expected value and variance o? is 
used to an LIT discrete-time system, of the impulse response h(n) = (0.5)"u(n) 
and produces the WSS output signaly(7). Compute the power spectrum P,,(@) 
of the output. 


SOLUTION: 


Py(@) = H(e)H(e")P,.(@). 


But P.,.(@)= 072, since E[x]=0. 
Moreover: 


l(c”) = H(e)H(e"”) 
1 1 
~ 1-0.5e” 1-05e 
7 1 
~ 1-0.5e” —0.5e7/” + (0.5)? 
i 
~ 1.25-cosa’ 


Finally, by combining the above relations, it is: 


x 


o)=————_.. 
1.25-—cos@ 


Py ( 


A.II.21 Stochastic WSS signal x() with the expected value E[x(n)]=2 and second sta- 
tistical moment E[x?]=7 is applied to an LTI discrete-time system, producing 
the stochastic signal y(n). The realization structure of the system is illustrated 
in the following figure. Calculate the expected value and the variance of the 
output signal. 
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SOLUTION: For the input signal, it is: 


u[x(n)] = 2, 
o? = E[x*]—-E[x(n)] =7 —2° =3. 


x 


Also, for the system, it is: 


1 1 
yln)=—, tn) +xn—-T)—, xln—2) 


© Y(z)=- ; X(z)+z'X(z)- 52 °X(2) 
ay H(z)= 1a - 12 
X(z) 2 2 


By taking the inverse z transform of H(z) , h(n) is derived: 


2 


n=0 


HFA eH D2 HDHD ARG 


For the output, it is: 


aLy(n)]= BLx(m)] Y* h(n) = Len (0)+H()+H)1=2| ~2 41-3 ]=0, 


n=—00 


5S eyes tagal |o 
o2=02 ¥ |h(n)| -3[d+14 a5 


n=—0o 


Bibliography 


Abromowitz, M. and Stegun, I., Handbook of Mathematical Functions. Dover, 1965. 

Adali, J. and Hayken, S., Adaptive Signal Processing: Next Generation Solutions. Wiley, 2010. 

Antoniou, A., Digital Filters: Analysis, Design, and Applications, 2nd Edition. McGraw-Hill, 1993. 

Bickel, P.-J. and Doksum, K.-A., Mathematical Statistics: Basic Ideas and Selected Topics. Holden-Day, 
1977. 

Bode, H. and Shannon, C., “A simplified derivation of linear least squares smoothing and prediction 
theory,” Proc. of the Institute of Radio Engineers (IRE), vol. 38, pp. 417-425, April 1950. 

Bose, T., Digital Signal and Image Processing. Wiley, 2003. 

Brillinger, D.-R., Time Series: Data Analysis and Theory. Springer-Verlag, 1981. 

Cavicchi, T., Digital Signal Processing. Wiley, 2000. 

Chaparro, L., Signals and Systems Using MATLAB. Elsevier, 2011. 

Chassaing, R., Digital Signal Processing with the C6713 and C6416 DSK. Wiley, 2004. 

Chassaing, R. and Reay, D., Digital Signal Processing and Applications with the TMS320C6713 DSK. 
Wiley-IEEE, 2008. 

Chen, C-T., Digital Signal Processing. Oxford University Press, 2001. 

Clark, C., LabVIEW Digital Signal Processing. McGraw-Hill, 2005. 

Corinthios, M., Signals, Systems, Transforms, and Digital Signal Processing with MATLAB. CRC Press, 
2009. 

Cristi, R., Modern Digital Signal Processing. Thompson, 2004. 

Davenport, W. and Root, W., An introduction to the theory of random signals and noise. IEEE Press 
(reprint of 1958 McGraw-Hill edition), 1987. 

Eaton, M.-L., Multivariate Statistics—A Vector Space Approach. Wiley, 1983. 

Gelman, A., Carlin, J.-B., Stern, H.-S., and Rubin, D.-B., Bayesian Data Analysis. Chapman and Hall/ 
CRC, 1995. 

Gentleman, W.M. and G. Sande, “Fast fourier transforms—For fun and profit,” in AFIPS Conference 
Proceedings, Nov. 1966, vol. 29, pp. 563-578. 

Golub, G.-H. and Van Loan, C.-F., Matrix Computations, 2nd Edition. The Johns Hopkins University 
Press, 1989. 

Graybill, F-A., Matrices with Applications in Statistics. Wadsworth Publishing Co., 1983. 

Grover, D. and Diller, J., Digital Signal Processing. Prentice Hall, 1999. 

Hayes, M., Schaum's Outline of Digital Signal Processing. Schaum, 1999. 

Hayes, M., Statistical Digital Signal Processing and Modeling. Wiley, 1996. 

Haykin, S., Array Signal Processing. Prentice-Hall, 1985. 

Heck, B. and Kamen, E., Fundamentals of Signals and Systems Using the Web and MATLAB. Prentice 
Hall, 2007. 

Helstrom, C., Elements of signal detection and estimation. Prentice-Hall, 1995. 

Hollander, M. and Wolfe, D.-A., Nonparametric statistical methods, 2nd Edition. Wiley, 1991. 

Ifwachor, E. and Jervis, B., Digital Signal Processing, 2nd Edition. Addison Wesley, 2001. 

Ingle, V. and Proakis, J., Digital Signal Processing Using MATLAB. Cengage Learning, 2007. 

Jackson, L., Digital Filters and Signal Processing. Kluwer, 1989. 

Johnson, N.-L., Kotz, S., and Balakrishnan, A., Continuous Univariate Distributions: Vol. 2. Wiley, 1995. 

Kariya, T. and Sinha, B.-K., Robustness of Statistical Tests. Academic Press, 1989. 

Kassam, S. and Thomas, J., Nonparametric detection—Theory and Applications. Dowden, Hutchinson 
and Ross, 1980. 

Kay, S.-M., Statistical Estimation. Prentice-Hall, 1991. 

Kehtarnavaz, N. and Kim, N., Digital Signal Processing System Design Using LabVIEW. Newnes, 2005. 

Kuo, S. and Gan, W.-S., Digital Signal Processing. Prentice Hall, 2004. 


543 


544 Bibliography 


Lehmann, E.-L., Testing Statistical Hypotheses. Wiley, 1959. 

Lutovac, M., Tosic, D.,and Evans, B., Filter Design for Signal Processing, Using MATLAB and Mathematica. 
Prentice Hall, 2001. 

Lyons, R., Understanding Digital Signal Processing. Prentice Hall, 2004. 

Madisetti, V., The Digital Signal Processing Handbook, 2nd Edition. CRC Press, 2009. 

McClellan, J., Schafer, R., and Yoder, M., Signal Processing First. Prentice Hall, 1998. 

Mendel, J.-M., Lessons in Estimation for Signal Processing, Communications, and Control. Prentice-Hall, 

1995. 

Mersereau, R. and Smith, M., Digital Filtering. Wiley, 1993. 

Mitra, S., Digital Signal Processing Laboratory Using MATLAB. McGraw-Hill, 1999. 

Mitra, S., Digital Signal Processing: A Computer-Based Approach, 4th Edition. McGraw-Hill, 2011. 

Moon, T.-K. and Stirling, W.-C., Mathematical Methods and Algorithms for Signal Processing. Prentice 

Hall, 2000. 

Moon, T. and Stirling, W., Mathematical Methods and Algorithms for Signal Processing. Prentice Hall, 

2000. 

Muirhead, R.-J., Aspects of Multivariate Statistical Theory. Wiley, 1982. 

Oppenheim, A.-V. and Willsky, A.-S., Signals and Systems. Prentice-Hall, 1983. 

Oppenheim, A. and Schafer, R., Digital Signal Processing, 2nd Edition. Prentice Hall, 1999. 

Oppenheim, A. and Schafer, R., Discrete-Time Signal Processing, 3rd Edition. Prentice Hall, 2010. 

Orfanidis, S., Signal Processing. Prentice Hall, 1996. 

Papoulis, A., Probability, Random Variables, and Stochastic Processes, 3rd Edition. McGraw Hill, 1991. 

Parhi, K., VLSI Ditial Signal Processing and Implementation. Wiley, 1995. 

Parks, T. and Burrus, S., Digital Filter Design. Wiley, 1987. 

Phillips, C., Parr, J., and Riskin, E., Signals, System, and Transforms. Prentice Hall, 2003. 

Poral, B., A Course in Digital Signal Processing. Wiley, 1997. 

Proakis J.-G. and Manolakis, D.-G., Digital Signal Processing: Principles, Algorithms, and Applications. 
Prentice-Hall, 1996. 

Proakis, J. and Manolakis, D., Digital Signal Processing, 4th Edition. Prentice Hall, 2007. 

Rao, P., Signals and Systems. McGraw-Hill, 2008. 

Saben, W.E., Discrete-Time Signal Analysis and Design. Wiley, 2008. 

Scharf, L.-L., Statistical Signal Processing: Detection, Estimation, and Time Series Analysis. Addison- 
Wesley, 1991. 

Schilling, R. and Harris, S., Fundamentals of Digital Signal Processing Using MATLAB. Thomson, 2005. 

Schlichtharle, D., Digital Filters. Springer, 2000. 

Schuler, C., and Chugani, M., Digital Signal Processing. McGraw-Hill, 2004. 

Searns, S. and Hush, D., Digital Signal Processing with Examples in MATLAB. CRC Press, 2011. 

Shannon, C., “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27(3), 
pp. 379-423, 1948. 

Singh, A. and Srinivansan, S., Digital Signal Processing. Thompson, 2004. 

Smith, S., The scientists and engineer's guide to digital signal processing. www.DSPguide.com. 

Tanner, M.-A., Tools for Statistical Inference; Methods for the exploration of posterior distributions and 
likelihood functions. Springer-Verlag, 1993. 

Taylor, F., Digital Filter Design Handbook. Marcel Dekker, 1983. 

Taylor, F. and Mellott, J., Hands-On Digital Signal Processing. McGraw-Hill, 1995. 

Van-Trees, H.-L., Detection, Estimation, and Modulation Theory: Part III. Wiley, 2001. 

Veloni, A., Miridakis, N., and Boucouvala, E., Digital and Statistical Signal Processing. CRC, 2018. 

Whalen, A.-D., Detection of Signals in Noise, 2nd Edition. Academic Press, 1995. 

Widrow, B. and Kollar, I., Quantization Noise. Cambridge, 2008. 

Williams, A. and Taylor, F., Electronic Filter Design Handbook, 4th Edition. McGraw-Hill, 2006. 

Yates, W., Digital Signal Processing. CSP, 1989. 

Zelniker, G. and Taylor, F., Advanced Digital Signal Processing. Dekker, 1994. 


Index 


Page numbers followed by f and t indicate figures and tables, respectively. 


A 
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average risk minimization, 488-489 
disadvantages, 501 
minimax Bayes detector, 491-493, 492f 
minimum probability of error test, 490 
optimal Bayes test, 489-490 
performance evaluation of Bayes likelihood 
ratio test, 490-491, 491f 
priori probabilities, 488 
Bayes estimator, 407 
Bayesian estimation (examples), 413-420 
Gaussian signal 
estimation of, 415-417, 415f, 416f 
sign estimation, 419-420 
size estimation, 417-419, 418f 
uniform PDF amplitude estimation, 413-415, 
414f 
Bayes likelihood ratio test, performance 
evaluation of, 490-491, 491f 
Bayes risk criterion, 422 
Bayes risk function, 407 
Beta distribution, 391-392 
BIBO stable (Bounded Input-Bounded Output 
stable), 58, 58f 
Bilateral z-transform, 116 
Bilinear (Tustin) method, digital IIR filters 
design, 297-298, 298f 
Binary codes, 10 
Binary likelihood ratios, 398 
Blackman window, 314, 315f 
Block diagrams 
branches, 179 
implementation components, 179, 180f 
nodes, 179 
realization structures, discrete-time systems, 
179-180, 180f 
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Branch(es) 
block diagrams, 179 
SFGs, 188, 188f 
‘Butterfly’ graph, FFT, 224, 224f 


C 


Cartesian form, z-transform, 113 
Casual discrete systems, 57-58 
Cauchy distribution, 391 
position parameters of, minimal sufficiency 
of, 402 
Cauchy’s formula, 123 
Cauchy’s residue theorem, 122 
Causality 
of discrete-time systems, 124-125 
CDF, see Cumulative distribution function 
(CDF) 
Central limit theorem (CLT), 387 
Chebychev inequality, 423, 424 
Chi-squared mixed distribution, 389-390 
Chi-squared RV distribution, 388 
non-central, 389 
Chunking method, 60-61 
Circular convolution 
defined, 219 
DFT of, 219, 219f 
Circular shift, DFT of, 218 
in frequency domain, 218 
in time domain, 218 
CME, see Conditional mean estimator (CME) 
Code, 9 
Codeword, 9 
Coding 
analog signals, 9-10, 9f—-10f 
defined, 9 
voice coding system, 10, 10f 
Commutativity property, convolution, 60, 60t 
Complex integration method 
inverse z-transform calculation, 123-124 
Complex system analysis, 58 
Composite hypothesis, signal detection, 485 
Computational complexity, 181 
Conditional mean estimator (CME), 407-408, 
408f 
minimum mean absolute error estimation, 409 
MSE estimation, 407-408 
MUE estimation, 412 
Conjugation of complex sequence, z-transform, 
120 
Connected systems 
feedback connection of systems, 126f 
systems connected in parallel, 126f 
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systems connected in series, 126f 
transfer function of, 126, 126f-127f 
Constant (multiple non-random) parameters, 
estimation of, 441 
Cramer-Rao (CR) matrix bound in 
covariance matrix, 442-446 
criteria for estimator performance 
evaluation:, 441 
maximum likelihood vector estimation, 
447-451 
vector estimation methods, 446 
Constant (non-random) parameters, estimation 
of, 422-423, 423f 
Cramer-Rao Bound (CRB) in estimation 
variance, 433-440 
maximum likelihood (ML), scalar estimators 
for, 429-433, 430f, 431f 
method of statistical moments for scalar 
estimators, 426-429, 428f 
scalar estimation criteria, 423-425, 
424f-425f 
Continuous signal, 7 
Continuous-time signal, 11 
Convolution, 59-62, 60f, 60t 
associativity property, 60, 60t 
circular, DFT, 219, 219f 
commutativity property, 60, 60t 
distributivity property, 60, 60t 
fast, FFT of, 228 
overlap and add method, 228-229, 228t 
overlap and save method, 229 
identity property, 60, 60t 
matrix, use of, 60, 60f 
periodic, DFS, 215 
properties, 60t 
of two finite-length signals, 60-61, 60f 
z-transform, 119 
Correlation 
discrete-time signals, 63-64 
of two sequences, z-transform, 121 
Covariance matrix 
Cramer-Rao (CR) matrix bound in, 442-446 
Cramer-Rao bound (CRB) 
defined, 433-434 
in estimation variance, 433-440 
examples, 439-440 
predictions for non-random nuisance 
parameters, 453-454 
scalar CRB, properties, 436-439 
Cramer-Rao (CR) matrix bound, in covariance 
matrix, 442-446 
proof of, 445-446 
properties, 443-445 
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CRB, see Cramer-Rao bound (CRB) 

Cross-correlation 
discrete-time signals, 63-64 
properties, 64 

Cumulative distribution function (CDF) 
of Gaussian RV, 384-385 

Cut-off frequency, 11 


D 


DAG, see Digital to analog converter (DAC) 
DCT, see Discrete cosine transform (DCT) 
Decimation in frequency (DIF) technique, 227 
Decimation in time (DIT) process, 224, 226, 227 
Decision function, signal detection, 486-487, 
486f, 487f 
Deconvolution, 62 
Delay component, block diagram, 179, 180f 
Dependent variable 
conversion of, 54 
Difference equations (D.E.), 113 
discrete-time systems, 64—65 
solution via z-transform, 113-114, 114f 
Digital filters; see also Analog filters 
advantages, 287-288 
band-pass filter, 289, 289f 
band-stop filter, 289, 289f 
design specifications, 288-290, 289f-290f 
digital FIR filters, 288 
digital IIR filters, 288 
frequency transformations, 301-303 
digital IIR filters, design, 290-291 
direct methods, 300-301 
indirect methods, 292-300 
FIR filters, 303-304 
stability of, 307 
FIR filters, design, 307 
comparison of methods, 319, 319t 
optimal equiripple FIR filter design, 
317-319, 318f 
using frequency sampling method, 
309-311 
using window method, 311-317, 311f, 312t, 
313f-316f, 317t 
FIR linear phase filters, 304-307 
high-pass filter, 289, 289f 
low-pass filter, 289, 289f 
moving average filters, 307-308, 309f 
overview, 287-288 
pass band, 289, 290f 
signal filtering process using, 287, 288f 
stop band, 290, 290f 
transition band, 290, 290f 
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types of, 288, 289f 
vs. analog filters, 287 
Digital FIR filters, 288; see also Finite impulse 
response (FIR) filters 
Digital IIR filters, 288; see also Infinite impulse 
response (IIR) filters 
design, 290-291 
direct methods, 291, 300-301 
indirect methods, 291, 292-300 
direct design methods 
design of H(e”)? method, 300-301 
h[n], method of calculating, 301 
frequency transformations, 301-303 
indirect design methods, 291, 292-300 
backward difference method, 295-296, 295f 
bilinear (Tustin) method, 297-298, 298f 
forward difference method, 296 
impulse invariant method, 292 
matched pole-zero method, 299-300 
step invariant method (or z-transform 
method with sample and hold), 
293-294, 293f-294f 
optimization methods, 291 
Digital signal processing, 4f 
advantages, 4-5 
analog signal, 3-4, 4f 
applications, 3, 4—5, 6 
objectives, 3 
overview, 3 
Digital signal processor (DSP), 4, 5, 287 
Digital signal(s), 45 
analog signal conversion into, 5 
coding, 9-10, 9f-10f 
quantization, 7-9, 7f-8f 
sampling, 5-7, 6f 
defined, 3 
Digital to analog converter (DAC), 4, 5, 6, 287 
features, 4 
Digital voice coding system, 10, 10f 
Digitization 
analog signals, 5 
coding, 9-10, 9f-10f 
quantization, 7-9, 7f-8f 
sampling, 5-7, 6f 
defined, 5 
Direct methods, digital IIR filters design 
design of H(e/”)? method, 300-301 
h[n], method of calculating, 301 
Discrete cosine transform (DCT), 212, 229-231, 
231f 
Discrete Fourier series (DFS), 211, 214-216 
coefficients of, 215 
periodic convolution, 215 
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relation of components and DTFT over a 
period, 216 
Discrete Fourier transform (DFT), 211, 216-220; 
see also Fast Fourier transform (FFT) 
circular convolution, 219, 219f 
circular shift, 218 
complex multiplications for calculations, 
227t 
linearity, 218 
multiplication of sequences, 220 
N-point DFT, 217 
Parseval’s theorem, 220, 220t 
properties, 218-220 
sampling at DTFT frequency, 217, 217f 
time dependent, 216 
Discrete likelihood ratios, 398 
Discrete-time Fourier transform (DTFT), 211, 
212, 213t-214t 
defined, 212 
of fundamental signals, 213t 
properties, 214t 
relation of DFS components and, 216 
sampling at DTFT frequency, 217, 217f 
Discrete-time moving average system, 55 
Discrete-time signals, 54f; see also Discrete-time 
systems 
autocorrelation, 63-64 
correlation, 63-64 
cross-correlation, 63-64 
defined, 45, 46f 
energy of, 53 
even and odd, 51-53, 52f 
exponential function, 48-50, 49f-50f 
impulse function, 45, 46f, 46t 
power of, 53 
ramp function, 47, 48f 
sinusoidal sequence, 50-51, 51f 
transfer function of, 124 
unit rectangular function (pulse function), 
48, 48f 
unit step function, 47, 47f 
Discrete-time systems, 54-55, 54f; see also 
Discrete-time signals 
additivity in, 55, 56f 
block diagrams, 179-180, 180f 
casual, 57-58 
categories, 55-58 
causality of, 124-125 
convolution, 59-62, 60f, 60t 
associativity property, 60, 60t 
commutativity property, 60, 60t 
distributivity property, 60, 60t 
identity property, 60, 60t 
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matrix, use of, 60, 60f 
properties, 60t 
of two finite-length signals, 60-61, 60f 
deconvolution, 62 
difference equations, 64-65 
discrete-time moving average system, 55 
examples, 55 
of finite impulse response, 65-66 
homogeneity in, 55, 56f 
ideal discrete time delay system, 55 
implementation, 179 
using hardware, 179 
using software, 179 
impulse response, 59 
invertible, 57 
linear, 55-56, 56f 
with memory, 57 
overview, 179 
realization structure, 181-183, 181f-182f 
FIR discrete systems, 186-187, 186f, 187 
IIR discrete systems, 183-186, 184f-185f 
signal flow graphs (SFGs), 188, 188f-189f 
Mason’s gain formula, 189, 189f—-190f 
stability of, 125-126 
stable, 58, 58f 
system connections, 58 
mixed connection, 58, 59f 
parallel connection, 58, 59f 
in series connection, 58, 59f 
time-invariant, 56-57 
transfer function of, 127-128 
unstable, 58, 58f 
Discrete wavelets transform (DWT), 212, 
231-236, 232f 
Distribution(s); see also Gaussian distribution 
reproducing, 392 
Distributivity property, convolution, 60, 60t 
Division method 
inverse z-transform calculation, 122 
DSP, see Digital signal processor (DSP) 
DTFT, see Discrete-time Fourier transform 
(DTFT) 
Duty cycle, 19 
DWT, see Discrete wavelets transform (DWT) 
DWT coefficients, 232 
Dynamic systems, 57 
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8-point FFT algorithm, 224-227, 225f-227f 
Energy 

of discrete-time signals, 53 
Even discrete-time signal, 51-53, 52f 
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Expected value, 393-395 
of Gaussian distribution, minimal 
sufficiency of, 400-401 
and variance of Gaussian distribution, 
minimal sufficiency of, 401-402 
Exponential distributions category, statistical 
sufficiency, 402-404 
PDF and, 404 
Exponential function, 48-50, 49f-50f 
Exponential sequence 
multiplication by, 120 


F 


Fast convolution, FFT of, 228 
overlap and add method, 228-229, 228t 
overlap and save method, 229 
Fast Fourier transform (FFT), 212, 221; see also 
Discrete Fourier transform (DFT) 
‘butterfly’ graph, 224, 224f 
complex multiplications for calculations, 
227t 
decimation in frequency (DIF) technique, 
227 
decimation in time (DIT) process, 224, 226, 
227 
8-point FFT algorithm, 224-227, 225f-227f 
equations, 221-227, 224f—226f, 227t 
fast convolution, 228-229 
Fourier transform estimation through, 229 
fundamental equations of radix-2 FFT 
algorithm, 223 
IDFT computation using FFT, 228 
iterative FFT algorithm, 223-224, 224f 
Feedback connection of systems, 126f 
Final value theorem, 121 
Finite energy signals, 53 
Finite impulse response (FIR) 
discrete-time systems of, 65-66 
Finite impulse response (FIR) filters, 288, 
303-304; see also Digital FIR filters 
design, 307 
comparison of methods, 319, 319t 
optimal equiripple FIR filter design, 
317-319, 318f 
using frequency sampling method, 
309-311 
using window method, 311-317, 311f, 312t, 
313f-316f, 317t 
ideal impulse responses, 312t 
stability of, 307 
Finite-length signals 
convolution of, 60-61, 60f 
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Finite power signals, 53 
Finite word length effects, 181 
FIR, see Finite impulse response (FIR) 
FIR discrete systems, implementation 
structures, 186-187 
in cascade form, 187, 187f 
in direct form I, 186, 186f 
FIR filters, see Finite impulse response (FIR) 
filters 
FIR linear phase filters, 304-307 
Fisher-Cochran theorem, 392-393, 394 
Fisher factorization, 397 
Fisher-Snedecor F-distribution, 390-391 
Formula tables 
z-transform, 129, 129t-130t 
Forward difference method, digital IIR filters 
design, 296 
Forward path, SFGs, 188, 189f 
Fourier analysis, 211 
Fourier transform; see also Discrete Fourier 
transform (DFT); Fast Fourier 
transform (FFT) 
estimation through FFT, 229 
Frequency domain, 211 
DFT of circular shift in, 218 
Frequency domain analysis 
discrete cosine transform, 229-231, 231f 
discrete Fourier series (DFS), 214-216 
periodic convolution, 215 
relation of components and DTFT over a 
period, 216 
discrete Fourier transform (DFT), 216-220 
circular convolution, 219, 219f 
circular shift, 218 
linearity, 218 
multiplication of sequences, 220 
Parseval’s theorem, 220, 220t 
properties, 218-220 
sampling at DTFT frequency, 217, 217f 
discrete-time Fourier transform (DTFT), 212, 
213t-214t 
fast Fourier transform (FFT), 221 
equations, 221-227, 224f—226f, 227t 
fast convolution, 228-229 
IDFT computation using FFT, 228 
Fourier transform, estimation through FFT, 
229 
overview, 211-212 
wavelet transform, 231-236, 232f 
theory, 233-236, 235f 
Frequency sampling method 
FIR filters design using, 309-311 
Frequency transformations, IIR filters, 301-303 
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Frequentist approach, signal detection, 502 
defined, 502 
simple hypothese, case of, 502-505 

FSR, see Full scale range (FSR) 

Full scale range (FSR), 8 


G 


Gamma distribution, 388-389 
Gaussian distribution, 383-392 
beta distribution, 391-392 
Cauchy distribution, 391 
central limit theorem, 387 
chi-squared mixed distribution, 389-390 
chi-squared RV distribution, 388 
Fisher-Snedecor F-distribution, 390-391 
gamma distribution, 388-389 
minimal sufficiency of expected value and 
variance of, 401-402 
minimal sufficiency of expected value of, 
400-401 
multivariate, 385-387 
non-central chi-squared RV distribution, 389 
overview, 383-385 
reproducing distributions, 392 
student’s t-distribution, 390 
Gaussian models 
linear, affine estimator optimization for, 
462-463 
LMMSE optimization in, 477-478 
Gaussian random variable X (RV); see also 
Gaussian distribution 
cumulative distribution function (CDF), 
384-385 
independent and identically distributed 
(IID), 385 
probability density function (PDF) of, 383 
reproducing distribution, 392 
Gaussian signal, Bayesian estimation, 415-417, 
415f, 416f 
sign estimation, 419-420 
size estimation, 417-419, 418f 
Gramm-Schmidt orthonormalization method, 394 
Graphical technique 
convolution calculation, 61 


H 


Hamming window, 314, 315f 
Hanning window, 313, 314f 
Hardware 
discrete-time system implementation using, 
179 
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Heaviside formula, 122 

He”)? method, digital IIR filters design, 
300-301 

High-pass filter, 289, 289f; see also Digital filters 

H[n], method of calculating, digital ITR filters 
design, 301 

Holding time, 20 

Homogeneity, in discrete-time systems, 55, 
56f 


I 


IDCT, see Inverse discrete Cosine transform 
(IDCT) 
Ideal discrete time delay system, 55 
Ideal impulse responses, 312t 
Identical system, 54 
Identity property, convolution, 60, 60t 
IDFT, see Inverse discrete Fourier transform 
(IDFT) 
IFFT, see Inverse Fast Fourier Transform (IFFT) 
IIR, see Infinite impulse response (IIR) 
IIR discrete systems, implementation 
structures, 183-186 
in canonical form, 184-185, 185f 
in direct form I, 184, 184f 
intermediate stage, 184, 184f 
IIR filters, see Infinite impulse response (IIR) 
filters 
Impulse function, 45 
defined, 45, 46f 
properties, 46t 
Impulse invariant method, digital IIR filters 
design, 292 
Impulse responses, 59; see also Convolution; 
Deconvolution 
ideal, 312t 
Independent and identically distributed (ITD) 
Gaussian RVs, 385 
Independent variable 
conversion of, 54 
lag, 54 
Indirect methods, digital IIR filters design, 291, 
292-300 
backward difference method, 295-296, 295f 
bilinear (Tustin) method, 297-298, 298f 
forward difference method, 296 
impulse invariant method, 292 
matched pole-zero method, 299-300 
step invariant method (or z-transform 
method with sample and hold), 
293-294, 293f-294f 
Infinite impulse response (IIR), 66 
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Infinite impulse response (IIR) filters, 288; 
see also Digital IIR filters 
frequency transformations, 301-303 
Initial value theorem, 121 
Input node, SFGs, 188, 188f 
Input-output relation, of one ADC 3 digits, 
7-8, 8f 
Inverse discrete Cosine transform (IDCT), 230 
Inverse discrete Fourier transform (IDFT), 221 
computation, using FFT, 228 
Inverse Fast Fourier Transform (IFFT), 221 
Inverse z-transform, 121-122 
complex integration method, 123-124 
defined, 122 
partial fraction expansion method, 122-123 
power series expansion method (division 
method), 122 
Invertible discrete systems, 57 
Iterative FFT algorithm, 223-224, 224f 


K 


Kaiser window, 315-316, 316f 
Karush-Kuhn-Tucker method, 503 


L 


Lag, 63 
Laplace transforms, 113 
defined, 113-114 
to z-transform, 113-116, 114f 
Linear/affine estimator, see Affine estimator 
Linear discrete systems, 55-56, 56f 
Linear estimation; see also Parametric 
estimation 
affine functions, 455 
constant MSE minimization (linear and 
affine estimation), 455 
optimal constant estimator of scalar RV, 
456 
geometric interpretation (orthogonality 
condition and projection theorem), 
459 
affine estimator optimization for linear 
Gaussian model, 462-463 
minimum affine MSE estimation, 462 
minimum MSE linear estimation, 
460-461, 461f 
LMWLS optimization in Gaussian models, 
477-478 
non-statistical least squares technique 
(linear regression), 467-472, 468f, 470f, 
4726, 473£ 
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optimal affine vector estimator, 463-464 
examples, 464—467 
overview, 455 
scalar random variable 
optimal affine estimator of, 458-459 
optimal constant estimator of, 456 
optimal linear estimator of, 456-458 
superposition property of linear/affine 
estimators, 459 
theory of, 455 
of weighted LLS, 473-477 
Linear Gaussian model, affine estimator 
optimization for, 462-463 
Linearity 
DFT, 218 
z-transform, 118 
Linear least squares theory (LLS), 455 
Linear minimum mean square error (LMMSE), 
455 
estimator, 455, 457-458; see also Linear 
estimation 
optimization, in Gaussian models, 
477-478 
Linear projection theorem, 460-461, 461f 
Linear regression, 467-472, 468f, 470f, 472f, 
473f 
Linear time invariant (LTT) systems, 55, 57, 116, 
124; see also Discrete-time systems 
impulse response of, 59-60, 60t; see also 
Convolution 
Linear WSSE estimator, 477 
LMMSE, see Linear minimum mean square 
error (LMMSE) 
Loop 
SFGs, 188, 188f-189f 
Low-pass filter, 289, 289f; see also Digital filters 


M 


Mason, Samuel J., 188 
Mason’s gain formula, 189, 189f—-190f 
Matched pole-zero method, digital IIR filters 
design, 299-300 
Matrix 
convolution computation, 60, 60f 
covariance, Cramer-Rao (CR) matrix bound 
in, 442-446 
Maximum-likelihood (ML) method, 425 
scalar estimators for, 429-433, 430f, 431f 
vector estimation, 447-451 
Mean square error (MSE) 
conditional mean estimator (CME), 
407-408 
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constant MSE minimization (linear and 
affine estimation), 455 
optimal constant estimator of scalar RV, 
456 
estimation of, 407-409, 408f 
minimum affine MSE estimation, 462 
minimum MSE linear estimation, 460-461, 
461f 
Mean uniform error (MUE) 
CME, 412 
estimation of, 411-412, 411f-413f 
Memory, 181 
discrete systems with, 57 
Memory-based system, 57 
Memory-less system, 57 
MIMO systems (multiple inputs, multiple 
outputs), 54 
Minimal sufficiency, 399-402 
defined, 399-400 
examples, 400-402 
of expected value and variance of Gaussian 
distribution, 401-402 
of expected value of Gaussian distribution, 
400-401 
of position parameters of Cauchy 
distribution, 402 
proof, 400 
theorem, 400 
Minimax Bayes detector, 491-493, 492f 
Minimum mean absolute error 
CME, 409 
estimation of, 409-411, 410f 
MISO system, see Multiple inputs, single output 
(MISO) system 
Mixed connection, discrete-time systems, 58, 59f 
Mixed node, SFGs, 188, 188f 
ML method, see Maximum-likelihood (ML) 
method 
Model validation, 55 
Moving average filters, 307-308, 309f 
MSE, see Mean square error (MSE) 
Multiple hypotheses tests, signal detection, 
496-498, 497f 
average risk minimization, 498-501, 501f 
Bayes approach, disadvantages, 501 
priori probabilities, 498 
Multiple inputs, multiple outputs (MIMO 
systems), 54 
Multiple inputs, single output (MISO) system, 54 
Multiple non-random (constant) parameters, 
estimation of, 441 
Cramer-Rao (CR) matrix bound in 
covariance matrix, 442-446 
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criteria for estimator performance 
evaluation:, 441 
maximum likelihood vector estimation, 
447-451 
vector estimation methods, 446 
Multiplication of sequences, DFT, 220 
Multiplier component, block diagram, 179, 180f 
Multivariate Gaussian distribution, 385-387 


N 


Neyman-Pearson strategy, 502, 503 
Nodes 
block diagrams, 179 
Non-accurate reconstruction, of analog signals, 
21-22, 22f 
Non-central chi-squared RV distribution, 389 
Non-random nuisance parameters 
CRB predictions for, 453-454 
handling, 452-453 
Non-random (constant) parameters, estimation 
of, 422-423, 423f 
Cramer-Rao Bound (CRB) in estimation 
variance, 433-440 
maximum likelihood (ML), scalar estimators 
for, 429-433, 430f, 431f 
method of statistical moments for scalar 
estimators, 426-429, 428f 
scalar estimation criteria, 423-425, 424f-425f 
Non-statistical least squares technique (linear 
regression), 467-472, 468f, 470f, 472f, 
473f 
N-order difference equation, 64 
Normal distribution, 383; see also Gaussian 
distribution 
N-point DFT, 217 
Nuisance parameters, handling, 452 
CRB predictions for non-random, 453-454 
non-random, 452-453 
random, 452 
Nyquist frequency, 11 
Nyquist theorem, 211 


O 


Odd discrete-time signal, 51-53, 52f 

Optimal affine estimator, of scalar RV, 458-459 

Optimal affine vector estimator, 463-464 

Optimal Bayes test, 489-490 

Optimal constant estimator, of scalar RV, 456 

Optimal equiripple FIR filter design, 317-319, 
318f 

Optimal linear estimator, of scalar RV, 456-458 
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vector estimation methods, 446 
of non-random (constant) parameters, 
422-423, 423f 
Cramer-Rao Bound (CRB) in estimation 
variance, 433-440 
maximum likelihood (ML), scalar 
estimators for, 429-433, 430f, 431f 
method of statistical moments for scalar 
estimators, 426-429, 428f 
scalar estimation criteria, 423-425, 
424f-425f 
nuisance parameters, handling, 452 
CRB predictions for non-random, 
453-454 


Overlap and add method, 228-229, 228t Periodic convolution 
Overlap and save method, 229 DFS, 215 
Physical sampling, 17-20, 18f-19f, 20f 
P pulse format of sampled signal, 18, 19f 
signal spectra in, 19, 20f 
PAM, see Pulse amplitude modulation (PAM) Polar form, z-transform, 113 
PAM signal, 21 Pole-zero matching method, digital IIR filters 
Parallel connection, discrete-time systems, 58, design, 299-300 
59f Position parameters of Cauchy distribution, 
Parametric estimation; see also Linear minimal sufficiency of, 402 
estimation Power 
components, 405-406, 406f of discrete-time signals, 53 
of multiple non-random (constant) Power series expansion method 
parameters, 441 inverse z-transform calculation, 122 
Cramer-Rao (CR) matrix bound in Probability density function (PDF) 
covariance matrix, 442-446 Cauchy distribution, 391 
maximum likelihood vector estimation, chi-squared RV distribution, 388 
447-451 defined, 388-389 


exponential distribution category and, 404 
Fisher/Snedecor-F, 390-391 
gamma distribution, 388-389 
of Gaussian RV, 383 
non-central chi-squared RV distribution, 389 
student’s t-distribution, 390 
Pulse amplitude modulation (PAM), 21 
Pulse function (unit rectangular function), 
48, 48f 


Q 


Quantization, 10 
analog signals, 7-9, 7f-8f 


non-random, 452-453 defined, 7 
random, 452 error, 7, 7f 
overview, 405 noise, 8 


of random vector parameters, 421 
squared vector error, 421-422, 422f 
uniform vector error, 422 

of scalar random parameters, 406—407, 407f 
Bayesian estimation (examples), 413-420 
mean square error (MSE), 407-409, 408f 
mean uniform error (MUE), 411-412, 


R 


Radar application, signal detection and, 
479-481, 480f 
error analysis, 481-483, 481f-482f, 484f 
examples, 493-499 


411f-413f Radix-2 FFT algorithm, fundamental equations 
minimum mean absolute error, 409-411, of, 223 

410f Ramp function, 47, 48f 
Parseval’s theorem, 220, 220t Random nuisance parameters, handling, 452 
Partial fraction expansion method Random vector parameters, estimation of, 421 

inverse z-transform calculation, 122-123 squared vector error, 421-422, 422f 

Pass band, digital filters, 289, 290f uniform vector error, 422 
Path, SFGs, 188, 189f Realization structures, discrete-time systems, 
PDF, see Probability density function (PDF) 181-183, 181f-182f 
PDF amplitude (uniform), Bayesian estimation, block diagrams, 179-180, 180f 


413-415, 414f Cascade form, 181, 182f 
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direct form I, 181, 181f 
direct form II or canonical form, 181, 182f 
FIR discrete systems, 186-187, 186f, 187 
IIR discrete systems, 183-186, 184f-185f 
implementation, 179 
overview, 179 
parallel form, 181, 182f 
realization structure, 181-183, 181f-182f 
signal flow graphs (SFGs), 188, 188f-189f 
Mason’s gain formula, 189, 189f—-190f 
Real-time processing and reconstruction 
analog signals, 21 
Receiver operating characteristic (ROC) curve 
properties, 506 
for threshold testing, 506-516, 506f-509f 
examples, 510-516 
Reconstruction 
non-accurate reconstruction of analog 
signals, 21-22, 22f 
sinusoidal signals, 10-11 
Rectangular window, 313, 313f 
Reduction ratio (RR), statistical sufficiency and, 
396, 396t 
Region of convergence (ROC), 116 
s- and z-planes into, comparison of, 116-117, 
117f 
Residue theorem, 123 
ROC, see Region of convergence (ROC) 
ROC curve, see Receiver operating 
characteristic (ROC) curve 
RR, see Reduction ratio (RR) 


Ss 


Samples, variance of, 393-395 
Sampling, 5 
analog signals, 5—7, 6f 
example, 7 
frequency, 5, 6, 11 
and holding device, 20-21, 20f 
and holding method, 20-21, 20f 
period, 5, 10 
rate, 6 
sinusoidal signals, 10-11 
uniform, 5, 6f 
Sampling theorem, 11 
ideal sampling using impulses, 12f 
low-pass filter, 15, 16f 
proof and detailed discussion, 12-17, 12f-16f 
sampling system, 13, 14f 
signals, 13, 13f 
sinusoidal signals, 12-17, 12f-16f 
spectrum, 13-14, 14f-15f, 16f 
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Scalar estimation criteria, for non-random 
parameters, 423-425, 424f-425f 
Scalar estimators 
for maximum likelihood (ML), 429-433, 430f, 
431f 
statistical moments method for, 426-429 
Scalar random parameters, estimation of, 
406-407, 407f 
Bayesian estimation (examples), 413-420 
mean square error (MSE), 407-409, 408f 
mean uniform error (MUE), 411-412, 
411f-413f 
minimum mean absolute error, 409-411, 410f 
Scalar random variable 
optimal affine estimator of, 458-459 
optimal constant estimator of, 456 
optimal linear estimator of, 456-458 
superposition property of linear/affine 
estimators, 459 
Series connection, discrete-time systems in, 58, 59f 
SFGs, see Signal flow graphs (SFGs) 
Shifted impulse function, 45 
Signal(s); see also Analog signal(s); Digital 
signal(s) 
analog form, 3-4 
analog signal processing, 3-4, 4f 
defined, 3 
Signal detection, 479-516 
Bayes approach, 488 
average risk minimization, 488-489 
minimax Bayes detector, 491-493, 492f 
minimum probability of error test, 490 
optimal Bayes test, 489-490 
performance evaluation of Bayes 
likelihood ratio test, 490-491, 491f 
priori probabilities, 488 
decision function, 486-487, 486f, 487f 
Frequentist approach, 502 
defined, 502 
simple hypotheses, case of, 502-505 
general problem, 484—485, 485f 
multiple hypotheses tests, 496-498, 497f 
average risk minimization, 498-501, 501f 
Bayes approach, disadvantages, 501 
priori probabilities, 498 
radar application, 479-481, 480f 
error analysis, 481-483, 481f-482f, 484f 
examples, 493-499 
ROC curves for threshold testing, 506-516, 
506f-509f 
examples, 510-516 
properties, 506 
simple and composite hypotheses, 485 
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Signal filtering process, using digital filters, 287, 
288f 
Signal flow graphs (SFGs), 188, 188f-189f 
branch, 188, 188f 
canonical form implementation structure, 
189, 190f 
of difference equation, 189, 189f 
forward path, 188, 189f 
loop, 188, 189f 
Mason’s gain formula, 189, 189f-190f 
mixed node, 188, 188f 
path, 188, 189f 
sink node, 188, 188f 
source or input node, 188, 188f 
Signal spectra 
in physical sampling, 19, 20f 
Signal-to-noise ratio (SNR), 8-9, 19 
Simple hypothesis, signal detection, 485 
case of, 502-505 
Single-input, single-output system (SISO), 54 
Sink node, SFGs, 188, 188f 
Sinusoidal sequence, discrete-time, 50-51, 51f 
Sinusoidal signals 
reconstruction, 10-17 
sampling, 10-17 
sampling frequency, 11 
sampling period, 10 
sampling theorem, 11 
ideal sampling using impulses, 12f 
proof and detailed discussion, 12-17, 
12f-16f 
sampling system, 13, 14f 
signals, 13, 13f 
spectrum, 13, 14f 
SISO, see Single-input, single-output system 
(SISO) 
SNR, see Signal-to-noise ratio (GNR) 
Software 
discrete-time system implementation using, 
179 
Source node, SFGs, 188, 188f 
s-plane vs. z-plane, into region of convergence, 
116-117, 117f 
Squared vector error, estimation of, 421-422, 
422f 
Stability 
of discrete-time systems, 125-126 
Stable discrete systems, 58 
Stable system, 59f 
Static system, 57 
Statistical models 
expected value and variance of samples, 
393-395 
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Fisher-Cochran theorem, 392-393 
Gaussian distribution, 383-392 
beta distribution, 391-392 
Cauchy distribution, 391 
central limit theorem, 387 
chi-squared mixed distribution, 389-390 
chi-squared RV distribution, 388 
Fisher-Snedecor F-distribution, 390-391 
gamma distribution, 388-389 
multivariate, 385-387 
non-central chi-squared RV distribution, 
389 
overview, 383-385 
student’s t-distribution, 390 
overview, 383 
reproducing distributions, 392 
statistical sufficiency, 395-396 
exponential distributions category, 
402-404 
minimal sufficiency, 399-402 
PDF and exponential distribution 
category, 404 
reduction ratio and, 396, 396t 
sufficient condition, definition, 397-399 
Statistical moments method 
for scalar estimators, 426-429 
vector estimation through, 446 
Statistical sufficiency, 395-396 
exponential distributions category, 402-404 
minimal sufficiency, 399-402 
PDF and exponential distribution category, 
404 
reduction ratio and, 396, 396t 
sufficient condition, definition, 397-399 
Step invariant method (z-transform method 
with sample and hold), digital IIR 
filters design, 293-294, 293f-294f 
Step-size, 8 
Stop band, digital filters, 290, 290f 
Student’s t-distribution, 390, 393 
Sufficient condition 
binary likelihood ratios, 398 
defined, 397-399 
discrete likelihood ratios, 398 
examples, 397-399 
Fisher factorization, 397 
likelihood ratio trajectory, 399 
System analysis, 55 
z-transform in, 124 
causality of discrete-time systems, 
124-125 
stability of discrete-time systems, 
125-126 
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transfer function of connected systems, 
126, 126f-127f 
transfer function of discrete-time signal, 
124 
transfer function of discrete-time 
systems, 127-128 
System connections, discrete-time systems, 58 
mixed connection, 58, 59f 
parallel connection, 58, 59f 
in series connection, 58, 59f 
System identification, 55 
System modeling, 55 


T 


Time Dependent Discrete Fourier Transform, 
216 

Time domain 

DFT of circular shift in, 218 
Time-invariant discrete systems, 56-57 
Time reversal, 54 

mirroring, 54 

z-transform, 118-119 
Time scaling, 54 
Time shift, 54 

z-transform, 117-118 
Tracking time, 18 
Transfer function 

of connected systems, 126, 126f-127f 

defined, 124 

of discrete-time signals, 124 

of discrete-time systems, 127-128 
Transition band, digital filters, 290, 290f 
Triangular (Bartlett) window, 313, 314f 
Tustin (bilinear) method, digital IIR filters 

design, 297-298, 298f 


Ga 


MVU estimator, see Uniform minimum 
variance unbiased (UMVU) 
estimator 

Uniform minimum variance unbiased (UMVU) 

estimator, 424—425, 425f 

Uniform PDF amplitude, Bayesian estimation, 
413-415, 414f 

niform sampling, 5, 6f 

niform vector error, estimation of, 422 

nilateral z-transform, 116 

nit rectangular function (pulse function), 48, 
48f 

nit step function, 47, 47f 

nstable system, 59f 
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Vv 


Variance 
expected value and, of Gaussian 
distribution, minimal sufficiency of, 
401-402 
of samples, 393-395 
Vector estimation 
maximum likelihood, 447-451 
through statistical moments, 446 
Vector estimator, optimal affine, 463-464 


Ww 


Wavelets, defined, 232 
Wavelet transform, 212, 231-236, 232f 
theory, 233-236, 235f 
Weighted LLS, linear estimation of, 473-477 
Window functions 
Bartlett (triangular), 313, 314f 
Blackman, 314, 315f 
election, 316, 317t 
Hamming, 314, 315f 
Hanning, 313, 314f 
Kaiser, 315-316, 316f 
rectangular, 313, 313f 
Window method, FIR filters design using, 
311-317, 311f, 312t, 313f-316f, 317t 


Z 


Zero-order hold filter (ZOH), 293 
operation of, 293-294, 293f 
signal reconstruction with, 294, 294f 
ZOH, see Zero-order hold filter (ZOH) 
z-plane 
differentiation in, 119-120 
vs. s-planes into region of convergence, 
116-117, 117f 
z-Transform 
bilateral, 116 
Cartesian form, 113 
defined, 113 
formula tables, 129, 129t-130t 
of fundamental functions, 129t 
inverse, 121-122 
complex integration method, 123-124 
partial fraction expansion method, 122-123 
power series expansion method (division 
method), 122 
from Laplace transform to, 113-116, 114f 
method with sample and hold, digital ITR 
filters design, 293-294, 293f-294f 
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Polar form, 113 

properties, 130t 
conjugation of complex sequence, 120 
convolution, 119 
correlation of two sequences, 121 
differentiation in z-plane, 119-120 
initial and final value theorem, 121 
linearity, 118 
multiplication by exponential sequence, 

120 

time reversal, 118-119 
time shift, 117-118 

solution of D.E. via, 113-114, 114f 
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s- us. z-planes into region of convergence, 
116-117, 117f 
in system analysis, 124 
causality of discrete-time systems, 
124-125 
stability of discrete-time systems, 125-126 
transfer function of connected systems, 
126, 126f-127f 
transfer function of discrete-time signal, 
124 
transfer function of discrete-time 
systems, 127-128 
unilateral, 116 


